May 22 00:57:01.551981 np0000176101 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. May 22 00:57:02.990364 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.common.config [-] Logging enabled! May 22 00:57:02.990973 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.1.0.dev264 May 22 00:57:02.990973 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=104709) setup_logging /opt/stack/neutron/neutron/common/config.py:125}} May 22 00:57:02.991343 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... May 22 00:57:02.991443 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=104709) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} May 22 00:57:02.991562 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=104709) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} May 22 00:57:02.991562 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=104709) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} May 22 00:57:02.991916 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=104709) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} May 22 00:57:02.991916 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=104709) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} May 22 00:57:02.991916 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [-] Loading core plugin: ml2 May 22 00:57:03.255700 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.255863 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.255863 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.256416 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.256501 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.256567 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.394194 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.394194 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.394194 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.399947 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.400014 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.400066 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.453393 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.453488 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.453488 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.456031 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.456082 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.456154 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.505848 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.505941 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.506029 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.506029 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.506309 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558459 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558548 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558587 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558673 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558728 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558783 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558844 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558897 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.558941 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559013 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559073 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559133 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559194 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559280 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559351 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559396 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559471 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559536 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559592 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559660 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559736 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559834 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559878 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.559943 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.560285 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:03.560329 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:03.560400 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:03.560444 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:03.560482 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:03.560543 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:03.560767 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] May 22 00:57:03.561519 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] May 22 00:57:03.562103 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete May 22 00:57:03.562597 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) May 22 00:57:03.562647 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] May 22 00:57:03.562811 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) May 22 00:57:03.562900 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Project network_types: ['geneve'] May 22 00:57:03.563011 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] May 22 00:57:03.563520 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] May 22 00:57:03.563564 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] May 22 00:57:03.563669 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] May 22 00:57:03.600572 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] May 22 00:57:03.600686 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] May 22 00:57:03.600686 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. May 22 00:57:03.601256 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.601386 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:03.625000 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' May 22 00:57:03.625084 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete May 22 00:57:03.625084 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' May 22 00:57:03.625241 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] May 22 00:57:03.667404 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO dbcounter [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Registered counter for database neutron May 22 00:57:03.677161 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_db.sqlalchemy.engines [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=104709) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} May 22 00:57:04.068914 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG dbcounter [-] [104709] Writer thread running {{(pid=104709) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} May 22 00:57:04.225850 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Initializing driver for type 'local' May 22 00:57:04.225850 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Initializing driver for type 'vlan' May 22 00:57:04.331951 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] VlanTypeDriver initialization complete May 22 00:57:04.332140 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Initializing extension driver 'port_security' May 22 00:57:04.332140 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.extensions.port_security [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] PortSecurityExtensionDriver initialization complete May 22 00:57:04.332140 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Initializing extension driver 'qos' May 22 00:57:04.332374 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.plugins.ml2.extensions.qos [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QosExtensionDriver initialization complete {{(pid=104709) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} May 22 00:57:04.332374 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Initializing mechanism driver 'ovn' May 22 00:57:04.332374 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Starting OVNMechanismDriver May 22 00:57:04.333815 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > process before_spawn 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.333908 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > process after_init 55550000, True {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.333981 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > segment after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334058 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > segment after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334135 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > segment after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334244 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334324 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334399 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334468 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334544 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334619 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334704 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334771 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334871 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.334947 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > address_group after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.335015 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > address_group after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.335076 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > address_group after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.335272 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.335413 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.335580 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.335957 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.services.logapi.drivers.ovn.driver [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] OVN logging driver registered May 22 00:57:04.337120 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.plugins.ml2.plugin [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=104709) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} May 22 00:57:04.337347 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > port after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.337476 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > port after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.337476 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > port after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.337627 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > subnet after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.337747 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > subnet after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.337747 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > subnet after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.337889 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > network after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.337889 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > network after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338013 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > network after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338013 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338156 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338156 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338306 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338383 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338450 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338527 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > address_group after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338594 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > address_group after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338661 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > address_group after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.338729 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ML2 OVO RPC backend initialized. {{(pid=104709) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} May 22 00:57:04.339380 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.quota [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded quota_driver: . May 22 00:57:04.339465 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.plugin [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Modular L2 Plugin initialization complete May 22 00:57:04.339570 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading services supported by the core plugin {{(pid=104709) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:170}} May 22 00:57:04.339654 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.plugins.ml2.managers [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Got port-security extension from driver 'port_security' May 22 00:57:04.339866 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=104709) _load_service_plugins /opt/stack/neutron/neutron/manager.py:195}} May 22 00:57:04.339946 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading Plugin: ovn-router May 22 00:57:04.347237 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347351 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: port before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347418 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: port after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347503 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > port precommit_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347579 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: port before_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347655 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: port before_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347729 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347797 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347869 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.347969 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: subnet after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.348130 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Tracking information for resource: router configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:04.348227 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.quota.resource_registry [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Tracking information for resource: floatingip configured {{(pid=104709) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} May 22 00:57:04.348296 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.services.ovn_l3.plugin [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Starting OVNL3RouterPlugin May 22 00:57:04.348468 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.348629 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router before_create 55550100, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.348691 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_delete 55550100, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.348781 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_create 55550100, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.348841 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_update 55550100, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.351196 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.services.provider_configuration [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Service providers = [] {{(pid=104709) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:170}} May 22 00:57:04.351297 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.db.servicetype_db [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=104709) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:45}} May 22 00:57:04.352317 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_create 55549900, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352410 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router_interface after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352499 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > floatingip after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352568 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352654 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352728 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352807 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352887 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > floatingip after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.352968 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.353045 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.353124 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router after_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.353210 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.353297 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router after_delete 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.353377 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > router after_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.353464 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.services.service_base [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=104709) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} May 22 00:57:04.353567 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: > process after_init 55550001, True {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.353659 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=104709) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 22 00:57:04.353740 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading Plugin: auto_allocate May 22 00:57:04.354449 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: network precommit_update 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.354535 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron_lib.callbacks.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Subscribe: network precommit_create 55550000, False {{(pid=104709) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.354700 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=104709) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 22 00:57:04.354766 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading Plugin: tag May 22 00:57:04.356823 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=104709) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 22 00:57:04.356899 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading Plugin: timestamp May 22 00:57:04.357897 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=104709) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 22 00:57:04.357962 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading Plugin: network_ip_availability May 22 00:57:04.359121 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=104709) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 22 00:57:04.359234 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading Plugin: flavors May 22 00:57:04.359921 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=104709) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 22 00:57:04.360010 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading Plugin: revisions May 22 00:57:04.360778 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.manager [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=104709) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} May 22 00:57:04.360880 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_concurrency.lockutils [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.369s {{(pid=104709) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} May 22 00:57:04.361035 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=104709) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} May 22 00:57:04.361120 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Initializing extension manager. May 22 00:57:04.361348 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: __init__.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.361423 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: __pycache__ {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.361502 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: aap_reject_multicast.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.361717 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Allowed address pairs reject multicast extension" alias="aap-reject-multicast" description="Reject multicast addresses in Allowed Address Pairs" updated="2026-04-01T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.361884 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension aap-reject-multicast not supported by any of loaded plugins May 22 00:57:04.361958 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: address_group.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.362224 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.362355 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: address-group May 22 00:57:04.362438 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: address_scope.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.362667 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.362795 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: address-scope May 22 00:57:04.362863 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: admin_state_down_before_update.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.363044 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.363182 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins May 22 00:57:04.363283 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: agent.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.363518 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.363653 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: agent May 22 00:57:04.363726 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: agent_resources_synced.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.363898 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.364029 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension agent-resources-synced not supported by any of loaded plugins May 22 00:57:04.364103 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: agent_sort_key.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.364290 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Agents Sort Key" alias="agent_sort_key" description="Enabling the keyword sort_key for sorting in agents." updated="2013-02-03T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.364420 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension agent_sort_key not supported by any of loaded plugins May 22 00:57:04.364501 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: allowedaddresspairs.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.365009 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.365148 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: allowed-address-pairs May 22 00:57:04.365233 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: auto_allocated_topology.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.365427 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.365559 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: auto-allocated-topology May 22 00:57:04.365629 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: availability_zone.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.365875 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.366003 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: availability_zone May 22 00:57:04.366084 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: availability_zone_filter.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.366276 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.366399 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension availability_zone_filter not supported by any of loaded plugins May 22 00:57:04.366471 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: data_plane_status.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.366635 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.366764 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension data-plane-status not supported by any of loaded plugins May 22 00:57:04.366833 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: default_subnetpools.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.367014 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.367191 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: default-subnetpools May 22 00:57:04.367191 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: dhcpagentscheduler.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.367514 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.367654 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: dhcp_agent_scheduler May 22 00:57:04.367707 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: dns.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.367897 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.368030 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: dns-integration May 22 00:57:04.368080 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: dns_domain_ports.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.368291 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.368414 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: dns-domain-ports May 22 00:57:04.368484 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.368670 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.368794 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: dns-integration-domain-keywords May 22 00:57:04.368865 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: dvr.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.369060 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.369189 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension dvr not supported by any of loaded plugins May 22 00:57:04.369265 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: ecmp_routes.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.369449 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.369578 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension ecmp_routes not supported by any of loaded plugins May 22 00:57:04.369648 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: empty_string_filtering.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.369832 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.369962 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: empty-string-filtering May 22 00:57:04.370027 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: evpn.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.370221 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="EVPN" alias="evpn" description="Router extension to support EVPN Type-5 route advertisement" updated="2026-04-28T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.370347 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension evpn not supported by any of loaded plugins May 22 00:57:04.370418 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.370592 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.370722 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins May 22 00:57:04.370787 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.370962 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.371089 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins May 22 00:57:04.371159 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: external_net.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.371362 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.371492 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: external-net May 22 00:57:04.371562 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: extra_dhcp_opt.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.371735 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.371865 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: extra_dhcp_opt May 22 00:57:04.371934 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: extraroute.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.372104 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.372250 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: extraroute May 22 00:57:04.372316 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: extraroute_atomic.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.372504 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.372631 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: extraroute-atomic May 22 00:57:04.372700 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: filter_validation.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.372888 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.373015 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: filter-validation May 22 00:57:04.373079 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: fip_pf_description.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.373269 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.373491 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins May 22 00:57:04.373573 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: fip_pf_detail.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.373794 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.373917 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: floating-ip-port-forwarding-detail May 22 00:57:04.373989 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: fip_pf_port_range.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.374164 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.374312 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins May 22 00:57:04.374381 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: fip_port_details.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.374554 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.374684 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: fip-port-details May 22 00:57:04.374753 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: flavors.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.374953 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.375074 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: flavors May 22 00:57:04.375143 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.375419 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.375588 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins May 22 00:57:04.375641 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: floatingip_pools.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.375816 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.375941 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: floatingip-pools May 22 00:57:04.376009 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: ip_allocation.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.376182 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.376329 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension ip_allocation not supported by any of loaded plugins May 22 00:57:04.376399 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: ip_substring_port_filtering.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.376571 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.376697 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension ip-substring-filtering not supported by any of loaded plugins May 22 00:57:04.376766 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l2_adjacency.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.376936 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.377065 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension l2_adjacency not supported by any of loaded plugins May 22 00:57:04.377134 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.377471 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.377609 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: router May 22 00:57:04.377701 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_conntrack_helper.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.378148 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.378306 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension l3-conntrack-helper not supported by any of loaded plugins May 22 00:57:04.378369 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.378561 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.378694 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: enable-default-route-bfd May 22 00:57:04.378759 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.378948 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.379078 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: enable-default-route-ecmp May 22 00:57:04.379142 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_ext_gw_mode.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.379362 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.379484 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: ext-gw-mode May 22 00:57:04.379554 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_ext_ha_mode.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.379727 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.379854 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: l3-ha May 22 00:57:04.379923 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.380095 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.380241 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins May 22 00:57:04.380312 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_extra_gws.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.380484 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.380612 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: external-gateway-multihoming May 22 00:57:04.380682 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_flavors.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.380859 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.380981 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: l3-flavors May 22 00:57:04.381051 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_ndp_proxy.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.381299 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.381415 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension l3-ndp-proxy not supported by any of loaded plugins May 22 00:57:04.381489 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.381663 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.381821 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins May 22 00:57:04.381893 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: l3agentscheduler.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.382315 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.382447 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension l3_agent_scheduler not supported by any of loaded plugins May 22 00:57:04.382512 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: local_ip.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.382761 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.382893 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension local_ip not supported by any of loaded plugins May 22 00:57:04.382958 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: logging.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.383191 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.383345 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension logging not supported by any of loaded plugins May 22 00:57:04.383414 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: metering.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.383777 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.383912 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension metering not supported by any of loaded plugins May 22 00:57:04.383977 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.384168 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.384316 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins May 22 00:57:04.384382 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: multiprovidernet.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.384568 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.384693 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: multi-provider May 22 00:57:04.384761 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: netmtu.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.384944 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.385074 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: net-mtu May 22 00:57:04.385145 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: netmtu_writable.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.385339 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.385468 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: net-mtu-writable May 22 00:57:04.385539 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: network_availability_zone.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.385763 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.385886 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: network_availability_zone May 22 00:57:04.385954 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: network_ha.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.386126 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.386271 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension network_ha not supported by any of loaded plugins May 22 00:57:04.386341 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: network_ip_availability.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.386528 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.386662 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: network-ip-availability May 22 00:57:04.386731 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: network_ip_availability_details.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.386917 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.387048 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: network-ip-availability-details May 22 00:57:04.387115 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: network_segment_range.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.387392 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.387523 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension network-segment-range not supported by any of loaded plugins May 22 00:57:04.387585 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: pagination.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.387769 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.387838 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: pagination May 22 00:57:04.387908 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_device_profile.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.388084 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.388234 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: port-device-profile May 22 00:57:04.388300 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_hardware_offload_type.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.388481 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.388605 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-hardware-offload-type not supported by any of loaded plugins May 22 00:57:04.388674 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.388845 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.388974 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins May 22 00:57:04.389042 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_hints.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.389233 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.389364 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-hints not supported by any of loaded plugins May 22 00:57:04.389432 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_mac_address_override.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.389605 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.389761 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-mac-override not supported by any of loaded plugins May 22 00:57:04.389833 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_mac_address_regenerate.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.390004 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.390132 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: port-mac-address-regenerate May 22 00:57:04.390212 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_numa_affinity_policy.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.390389 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.390516 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: port-numa-affinity-policy May 22 00:57:04.390584 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.390759 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.390887 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: port-numa-affinity-policy-socket May 22 00:57:04.390956 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_resource_request.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.391132 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.391292 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-resource-request not supported by any of loaded plugins May 22 00:57:04.391350 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_resource_request_groups.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.391534 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.391661 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-resource-request-groups not supported by any of loaded plugins May 22 00:57:04.391904 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: port_trusted_vif.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.392042 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.392166 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-trusted-vif not supported by any of loaded plugins May 22 00:57:04.392253 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: portbindings.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.392426 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.392557 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: binding May 22 00:57:04.392628 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: portbindings_extended.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.392825 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.392956 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: binding-extended May 22 00:57:04.393017 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: portsecurity.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.393185 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.393330 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: port-security May 22 00:57:04.393399 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: project_id.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.393576 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.393655 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: project-id May 22 00:57:04.393747 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: providernet.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.393903 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.394031 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: provider May 22 00:57:04.394098 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qinq.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.394314 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x88a8 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.394436 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: qinq May 22 00:57:04.394505 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.395154 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.395318 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos not supported by any of loaded plugins May 22 00:57:04.395383 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_bw_limit_direction.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.395597 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.395728 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-bw-limit-direction not supported by any of loaded plugins May 22 00:57:04.395791 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.395973 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.396096 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins May 22 00:57:04.396165 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_default.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.396361 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.396482 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-default not supported by any of loaded plugins May 22 00:57:04.396548 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_fip.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.396719 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.396845 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-fip not supported by any of loaded plugins May 22 00:57:04.396914 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_fip_network_policy.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.397132 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.397227 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-fip-network-policy not supported by any of loaded plugins May 22 00:57:04.397298 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_gateway_ip.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.397468 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.397593 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-gateway-ip not supported by any of loaded plugins May 22 00:57:04.397663 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_port_network_policy.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.397857 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.397984 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-port-network-policy not supported by any of loaded plugins May 22 00:57:04.398053 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.398253 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.398381 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-pps-minimum not supported by any of loaded plugins May 22 00:57:04.398451 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.398637 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.398766 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins May 22 00:57:04.398834 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_pps_rule.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.399023 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.399151 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-pps not supported by any of loaded plugins May 22 00:57:04.399250 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_rule_type_details.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.399423 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.399552 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-rule-type-details not supported by any of loaded plugins May 22 00:57:04.399622 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_rule_type_filter.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.399791 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.399919 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-rule-type-filter not supported by any of loaded plugins May 22 00:57:04.399988 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: qos_rules_alias.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.400170 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.400319 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension qos-rules-alias not supported by any of loaded plugins May 22 00:57:04.400388 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: quota_check_limit.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.400558 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.400689 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: quota-check-limit May 22 00:57:04.400758 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: quota_check_limit_default.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.400927 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.401056 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: quota-check-limit-default May 22 00:57:04.401125 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: quotasv2.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.401478 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.401554 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: quotas May 22 00:57:04.401625 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: quotasv2_detail.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.402030 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.402104 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: quota_details May 22 00:57:04.402178 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: rbac.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.402557 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control project access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.402699 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: rbac-policies May 22 00:57:04.402771 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: rbac_address_group.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.402951 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.403086 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension rbac-address-group not supported by any of loaded plugins May 22 00:57:04.403150 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: rbac_address_scope.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.403362 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.403488 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: rbac-address-scope May 22 00:57:04.403560 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: rbac_security_groups.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.403736 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.403864 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: rbac-security-groups May 22 00:57:04.403935 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: rbac_subnetpool.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.404336 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.404336 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension rbac-subnetpool not supported by any of loaded plugins May 22 00:57:04.404336 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: revisionifmatch.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.404947 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.404947 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: revision-if-match May 22 00:57:04.404947 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: revisions.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.404947 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.405085 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: standard-attr-revisions May 22 00:57:04.405085 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: router_availability_zone.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.405251 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.405382 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: router_availability_zone May 22 00:57:04.405447 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: router_enable_snat.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.405626 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.405792 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: router-enable-snat May 22 00:57:04.405856 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: routerservicetype.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.406019 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.406148 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension router-service-type not supported by any of loaded plugins May 22 00:57:04.406246 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: security_groups_default_rules.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.406575 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.406709 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: security-groups-default-rules May 22 00:57:04.406772 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.406956 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.407086 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: security-groups-normalized-cidr May 22 00:57:04.407151 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: security_groups_port_filtering.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.407352 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.407473 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension port-security-groups-filtering not supported by any of loaded plugins May 22 00:57:04.407543 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: security_groups_remote_address_group.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.407715 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.407843 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: security-groups-remote-address-group May 22 00:57:04.407911 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.408083 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.408228 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: security-groups-rules-belongs-to-default-sg May 22 00:57:04.408305 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: security_groups_shared_filtering.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.408479 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.408614 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: security-groups-shared-filtering May 22 00:57:04.408685 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: securitygroup.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.409958 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.410103 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: security-group May 22 00:57:04.410173 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: segment.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.410444 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.410590 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension segment not supported by any of loaded plugins May 22 00:57:04.410655 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.410842 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.410977 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins May 22 00:57:04.411032 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: servicetype.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.411239 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.411371 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: service-type May 22 00:57:04.411442 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: sorting.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.411634 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.411701 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: sorting May 22 00:57:04.411772 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: standard_attr_segment.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.411946 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.412077 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension standard-attr-segment not supported by any of loaded plugins May 22 00:57:04.412147 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: standardattrdescription.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.412357 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.412488 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: standard-attr-description May 22 00:57:04.412558 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: stateful_security_group.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.412731 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.412862 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: stateful-security-group May 22 00:57:04.412932 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: stdattrs_common.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.413086 np0000176101 neutron-ovn-maintenance-worker[104709]: WARNING neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py May 22 00:57:04.413156 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.413363 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.413494 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: subnet-dns-publish-fixed-ip May 22 00:57:04.413562 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: subnet_external_network.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.413763 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.413894 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: subnet-external-network May 22 00:57:04.413964 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: subnet_onboard.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.414146 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.414297 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension subnet_onboard not supported by any of loaded plugins May 22 00:57:04.414355 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: subnet_segmentid_writable.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.414537 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.414670 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension subnet-segmentid-writable not supported by any of loaded plugins May 22 00:57:04.414741 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: subnet_service_types.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.414913 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.415041 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: subnet-service-types May 22 00:57:04.415111 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: subnetallocation.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.415435 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.415563 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: subnet_allocation May 22 00:57:04.415634 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.415827 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.415961 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: subnetpool-prefix-ops May 22 00:57:04.416030 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: tag_creation.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.416216 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.416353 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: tag-creation May 22 00:57:04.416425 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.416599 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.416729 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins May 22 00:57:04.416800 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: tagging.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.417362 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.417501 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: standard-attr-tag May 22 00:57:04.417556 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: timestamp.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.417779 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.417910 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: standard-attr-timestamp May 22 00:57:04.417990 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: trunk.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.418179 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.418317 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension trunk not supported by any of loaded plugins May 22 00:57:04.418387 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: trunk_details.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.418581 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.418692 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension trunk-details not supported by any of loaded plugins May 22 00:57:04.418761 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: uplink_status_propagation.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.418935 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.419064 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension uplink-status-propagation not supported by any of loaded plugins May 22 00:57:04.419133 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.419371 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.419446 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins May 22 00:57:04.419528 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loading extension file: vlantransparent.py {{(pid=104709) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} May 22 00:57:04.419703 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=104709) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} May 22 00:57:04.419831 np0000176101 neutron-ovn-maintenance-worker[104709]: INFO neutron.api.extensions [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Loaded extension: vlan-transparent May 22 00:57:04.422000 np0000176101 neutron-ovn-maintenance-worker[104709]: WARNING oslo_service.backend._threading.service [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Service MaintenanceWorker is not picklable with spawn; falling back to fork. Please make the service spawn-safe to avoid this fallback. May 22 00:57:04.422303 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Full set of CONF: {{(pid=104709) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} May 22 00:57:04.422363 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ******************************************************************************** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} May 22 00:57:04.422425 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Configuration options gathered from: {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} May 22 00:57:04.422497 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} May 22 00:57:04.422597 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} May 22 00:57:04.422665 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ================================================================================ {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} May 22 00:57:04.422762 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] agent_down_time = 75 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.422851 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] allow_automatic_dhcp_failover = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.422935 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] allow_bulk = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423105 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423193 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] api_extensions_path = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423302 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] api_paste_config = api-paste.ini {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423395 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] api_workers = 4 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423484 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] auth_strategy = keystone {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423574 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] base_mac = fa:16:3e:00:00:00 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423676 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] bind_host = 0.0.0.0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423765 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] bind_port = 9696 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423851 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] client_socket_timeout = 900 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.423941 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] config_dir = [] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424006 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424094 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] config_source = [] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424185 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] control_exchange = neutron {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424293 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] core_plugin = ml2 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424381 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] debug = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424467 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] default_availability_zones = [] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424619 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424709 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] dhcp_agent_notification = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424792 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] dhcp_agents_per_network = 1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424876 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] dhcp_lease_duration = 86400 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.424965 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] dhcp_load_type = networks {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425054 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] dns_domain = openstacklocal {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425139 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425239 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] enable_default_route_bfd = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425326 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] enable_default_route_ecmp = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425408 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] enable_dvr = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425492 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] enable_new_agents = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425577 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] enable_services_on_agents_with_admin_state_down = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425661 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] enable_snat_by_default = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425759 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] enable_traditional_dhcp = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425840 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] external_dns_driver = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.425923 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] filter_validation = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426013 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] global_physnet_mtu = 1430 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426077 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] graceful_shutdown_timeout = 60 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426194 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] host = np0000176101 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426313 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] host_dvr_for_dhcp = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426398 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] http_retries = 3 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426487 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] instance_format = [instance: %(uuid)s] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426578 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426661 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ipam_driver = internal {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426743 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ipv6_pd_enabled = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426826 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] l3_ha = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.426914 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427001 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] l3_ha_network_physical_name = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427113 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] l3_ha_network_type = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427170 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_color = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427278 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_config_append = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427368 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427452 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_dir = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427534 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_file = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427586 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_options = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427676 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_rotate_interval = 1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427764 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_rotate_interval_type = days {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427851 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] log_rotation_type = none {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427913 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.427975 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428063 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428150 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428229 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428314 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_allowed_address_pair = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428396 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_dns_nameservers = 5 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428483 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_header_line = 16384 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428563 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_l3_agents_per_router = 3 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428647 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_logfile_count = 30 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428730 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_logfile_size_mb = 200 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428811 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_routes = 30 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428892 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] max_subnet_host_routes = 20 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.428980 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] my_ip = 199.204.45.129 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429066 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] my_ipv6 = 2604:e100:1:0:f816:3eff:fe09:7061 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429150 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] network_auto_schedule = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429247 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] network_link_prefix = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429336 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429422 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] notify_nova_on_port_data_changes = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429508 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] notify_nova_on_port_status_changes = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429595 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] pagination_max_limit = -1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429678 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] periodic_fuzzy_delay = 5 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429792 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] periodic_interval = 40 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429875 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] publish_errors = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.429957 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rate_limit_burst = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430046 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rate_limit_except_level = CRITICAL {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430128 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rate_limit_interval = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430236 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] router_distributed = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430326 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rpc_resources_processing_step = 20 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430410 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rpc_response_max_timeout = 600 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430494 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rpc_response_timeout = 60 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430589 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rpc_state_report_workers = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430670 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] rpc_workers = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430753 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] send_events_interval = 2 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430927 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] service_plugins = ['ovn-router'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.430927 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] setproctitle = on {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431003 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] shell_completion = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431087 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] state_path = /opt/stack/data/neutron {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431176 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] syslog_log_facility = LOG_USER {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431281 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] tcp_keepidle = 600 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431370 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] transport_url = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431488 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] use_journal = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431661 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] use_json = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431661 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] use_stderr = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431747 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] use_syslog = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431795 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] vlan_qinq = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431849 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] vlan_transparent = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431917 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] wsgi_default_pool_size = 100 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.431998 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] wsgi_keep_alive = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.432082 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.432262 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] wsgi_server_debug = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.432262 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_concurrency.disable_process_locking = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432371 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432466 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.connection_string = messaging:// {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432555 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.enabled = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432644 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.es_doc_type = notification {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432733 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.es_scroll_size = 10000 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432819 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.es_scroll_time = 2m {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432904 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.filter_error_trace = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.432992 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.hmac_keys = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433079 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.sentinel_service_name = mymaster {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433172 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.socket_timeout = 0.1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433274 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.trace_requests = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433360 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler.trace_sqlalchemy = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433452 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler_jaeger.process_tags = {} {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433536 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler_jaeger.service_name_prefix = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433626 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] profiler_otlp.service_name_prefix = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433731 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433816 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433906 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_metrics.metrics_process_name = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.433996 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434082 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434178 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.enforce_new_defaults = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434379 np0000176101 neutron-ovn-maintenance-worker[104709]: WARNING oslo_config.cfg [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. May 22 00:57:04.434476 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.enforce_scope = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434565 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.policy_default_rule = default {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434661 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434753 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434842 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.434925 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435009 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435091 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435176 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435283 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_policy.remote_timeout = 60.0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435377 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435464 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] service_providers.service_provider = [] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435555 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.auth_section = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435648 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.auth_type = password {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435740 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.auth_url = https://199.204.45.129/identity {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435809 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.cafile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435897 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.certfile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.435983 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.collect_timing = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436066 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.default_domain_id = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436148 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.default_domain_name = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436248 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.domain_id = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436332 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.domain_name = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436423 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.endpoint_type = public {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436510 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.insecure = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436593 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.keyfile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436685 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.password = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436768 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.project_domain_id = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.436858 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.project_domain_name = Default {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437293 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.project_id = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437293 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.project_name = service {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437293 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.region_name = RegionOne {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437293 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.split_loggers = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437293 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.system_scope = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437487 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.timeout = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437487 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.trust_id = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437569 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.user_domain_id = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437614 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.user_domain_name = Default {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437797 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.user_id = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437797 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] nova.username = nova {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437890 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.auth_section = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.437988 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.auth_type = password {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438070 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.cafile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438152 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.certfile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438268 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.collect_timing = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438361 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.endpoint_type = public {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438445 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.insecure = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438527 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.keyfile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438609 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.region_name = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438693 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.split_loggers = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438776 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] placement.timeout = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438865 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.auth_section = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.438952 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.auth_type = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439034 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.cafile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439115 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.certfile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439215 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.collect_timing = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439305 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.connect_retries = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439389 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.connect_retry_delay = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439475 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.enable_notifications = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439560 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.endpoint_override = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439643 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.insecure = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439727 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.interface = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.439807 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.keyfile = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440193 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.max_version = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440193 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.min_version = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440193 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.region_name = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440193 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.retriable_status_codes = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440193 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.service_name = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440340 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.service_type = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440449 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.split_loggers = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440449 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.status_code_retries = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440535 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.status_code_retry_delay = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440615 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.timeout = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440698 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.valid_interfaces = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440781 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ironic.version = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440869 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] cli_script.dry_run = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.440963 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441049 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441136 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441241 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441328 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441412 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441500 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441586 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441676 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.hostname = np0000176101 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441771 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441861 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.441950 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442037 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442125 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442241 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442330 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442414 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442505 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442591 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442675 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442760 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442845 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.442931 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443014 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443098 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443183 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443286 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443371 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443456 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443542 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.ssl = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443634 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443723 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443807 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443892 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.443981 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444069 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.ssl_version = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444160 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444270 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_notifications.driver = [] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444356 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_notifications.retry = -1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444450 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444536 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_messaging_notifications.transport_url = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444624 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_reports.file_event_handler = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444710 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_reports.file_event_handler_interval = 1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444793 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_reports.log_dir = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444882 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.444971 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.default_quota = -1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445060 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445146 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_floatingip = 50 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445256 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_network = 100 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445342 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_port = 500 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445428 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_rbac_policy = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445511 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_router = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445597 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_security_group = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445681 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_security_group_rule = 100 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445778 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.quota_subnet = 100 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445863 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] QUOTAS.track_quota_usage = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.445956 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446039 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep.group = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446125 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep.helper_command = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446227 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep.log_daemon_traceback = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446330 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep.logger_name = oslo_privsep.daemon {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446417 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep.thread_pool_size = 16 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446501 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep.user = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446593 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446678 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_dhcp_release.group = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446770 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_dhcp_release.helper_command = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446847 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.446936 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447021 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_dhcp_release.thread_pool_size = 16 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447105 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_dhcp_release.user = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447193 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447293 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_ovs_vsctl.group = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447375 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_ovs_vsctl.helper_command = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447460 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447549 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447633 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447716 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_ovs_vsctl.user = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447808 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_namespace.capabilities = [21] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447887 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_namespace.group = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.447967 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_namespace.helper_command = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448050 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_namespace.log_daemon_traceback = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448138 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448244 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_namespace.thread_pool_size = 16 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448327 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_namespace.user = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448415 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_conntrack.capabilities = [12] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448497 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_conntrack.group = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448579 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_conntrack.helper_command = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448663 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_conntrack.log_daemon_traceback = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448751 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448836 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_conntrack.thread_pool_size = 16 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.448919 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_conntrack.user = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449010 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_link.capabilities = [12, 21] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449093 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_link.group = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449175 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_link.helper_command = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449276 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_link.log_daemon_traceback = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449365 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449450 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_link.thread_pool_size = 16 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449532 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] privsep_link.user = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449630 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449749 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449830 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] AGENT.use_helper_for_ns_read = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.449920 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] SECURITYGROUP.enable_ipset = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450009 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] SECURITYGROUP.enable_security_group = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450093 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] SECURITYGROUP.firewall_driver = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450180 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450300 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] experimental.ipv6_pd_enabled = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450405 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450484 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.external_network_type = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450576 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.mechanism_drivers = ['ovn'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450667 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.overlay_ip_version = 4 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450751 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.path_mtu = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450838 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.physical_network_mtus = {} {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.450932 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.project_network_types = ['geneve'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451021 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451116 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451232 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2_type_flat.flat_networks = ['public'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451330 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2_type_geneve.max_header_size = 38 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451424 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451520 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451610 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] network_log.burst_limit = 25 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451692 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] network_log.local_output_log_base = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451778 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] network_log.rate_limit = 100 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451868 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.asyncio_connection = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.451951 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.asyncio_slave_connection = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452042 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.backend = sqlalchemy {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452128 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.connection = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452227 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.connection_debug = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452320 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.connection_parameters = {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452404 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.connection_recycle_time = 3600 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452491 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.connection_trace = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452581 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.db_inc_retry_interval = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452664 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.db_max_retries = 20 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452748 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.db_max_retry_interval = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452831 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.db_retry_interval = 1 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.452916 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.max_overflow = 50 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453001 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.max_pool_size = 5 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453083 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.max_retries = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453172 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.mysql_sql_mode = TRADITIONAL {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453271 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.mysql_wsrep_sync_wait = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453353 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.pool_timeout = None {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453437 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.retry_interval = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453520 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.slave_connection = **** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453603 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.sqlite_synchronous = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453687 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.synchronous_reader = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453787 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] database.use_db_reconnect = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453883 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.bfd_min_rx = 1000 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.453968 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.bfd_min_tx = 100 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454052 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.bfd_mult = 3 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454136 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.broadcast_arps_to_all_routers = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454251 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.dhcp_default_lease_time = 43200 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454341 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454425 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.dns_records_ovn_owned = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454512 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.dns_servers = [] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454596 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.enable_distributed_floating_ip = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454680 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.fdb_age_threshold = 300 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454769 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ha_failover_strategy = normal {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454859 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.live_migration_activation_strategy = rarp {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.454944 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.localnet_learn_fdb = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455029 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.mac_binding_age_threshold = 300 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455119 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.neutron_sync_mode = log {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455214 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_dhcp4_global_options = {} {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455307 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_dhcp6_global_options = {} {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455399 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455488 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_metadata_enabled = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455580 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455668 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455763 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_nb_connection = ['ssl:199.204.45.129:6641'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455851 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.455937 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_router_indirect_snat = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456025 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456112 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456215 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_sb_connection = ['ssl:199.204.45.129:6642'] {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456313 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456397 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovs_create_tap = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456484 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovsdb_connection_timeout = 180 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456575 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovsdb_log_level = INFO {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456658 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovsdb_probe_interval = 60000 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456742 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456826 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.stateless_nat_enabled = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.456914 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457006 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] OVS.bridge_mac_table_size = 50000 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457091 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] OVS.igmp_flood = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457176 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] OVS.igmp_flood_reports = True {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457278 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] OVS.igmp_flood_unregistered = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457371 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] OVS.igmp_snooping_enable = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457456 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] OVS.ovsdb_timeout = 10 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457549 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457633 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn_nb_global.ignore_lsp_down = False {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457771 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.457824 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG cotyledon.oslo_config_glue [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] ******************************************************************************** {{(pid=104709) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} May 22 00:57:04.457880 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG oslo_service.backend._threading.service [None req-85621e88-34c6-4c25-9574-ce5fe30de4ed None None] Selected the multiprocessing context: and updated it in Cotyledon ServiceManager: {{(pid=104709) launch_service /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py:405}} May 22 00:57:04.488156 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=105475) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} May 22 00:57:04.488745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} May 22 00:57:04.488745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} May 22 00:57:04.488745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} May 22 00:57:04.488745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} May 22 00:57:04.488745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} May 22 00:57:04.488745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.488745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 4 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489986 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] bind_host = 0.0.0.0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489986 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] bind_port = 9696 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489986 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.489986 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490244 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490244 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490244 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490411 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490411 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490513 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490727 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490870 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490870 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.490974 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.491107 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.491169 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.491275 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.491378 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.491473 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.491653 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.491924 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492029 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492121 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492244 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492305 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492375 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492443 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492499 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.492847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] host = np0000176101 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494224 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494320 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494320 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494458 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494458 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494566 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494629 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494685 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494821 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494888 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.494944 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495018 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495077 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495131 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495222 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495222 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495351 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495407 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495471 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495533 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495533 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495617 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495657 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495696 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495856 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495856 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495951 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.495951 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496041 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496087 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496130 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496193 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496298 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 199.204.45.129 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496366 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = 2604:e100:1:0:f816:3eff:fe09:7061 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496443 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496533 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.496913 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497136 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497238 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497315 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497391 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497449 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497522 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497805 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497805 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497805 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497926 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497926 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.497999 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498088 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498133 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498253 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498320 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498390 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498445 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498517 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498580 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498654 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498716 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498778 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498853 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498926 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.498980 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.499035 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.499095 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.499141 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.499258 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.499319 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} May 22 00:57:04.499432 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.499525 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.499626 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.499714 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.499782 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.499854 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.499927 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.499996 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500072 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500145 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500243 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500307 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500382 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500478 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500546 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500611 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500789 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500885 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.500969 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501042 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501142 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501183 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501250 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501292 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501332 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501387 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501428 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501466 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501505 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501544 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501583 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501693 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501785 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501887 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501947 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501987 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://199.204.45.129/identity {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.501987 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502061 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502100 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502139 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502177 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502232 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502273 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502322 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502362 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502401 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502440 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502478 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502517 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502556 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502593 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502631 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502678 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502716 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502796 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502839 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502877 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502922 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502961 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.502999 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503037 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503175 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503175 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503175 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503310 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503310 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503310 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503423 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503423 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503423 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503525 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503525 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503603 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503603 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503603 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503708 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503708 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503787 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503787 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503858 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503858 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503936 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.503936 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504009 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504009 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504079 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504079 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504151 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504219 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504219 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504304 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504304 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504375 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504375 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504442 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504442 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504524 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504571 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504616 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504661 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504699 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504760 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504803 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504846 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504885 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000176101 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504928 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.504995 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505049 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505090 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505128 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505173 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505226 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505276 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505276 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505349 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505395 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505440 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505440 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505518 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505518 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505591 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505591 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505663 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505663 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505945 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505945 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.505945 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506057 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506057 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506057 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506161 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506161 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506161 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506294 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506294 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506373 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506373 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506445 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506445 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506522 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506558 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506628 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506671 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506727 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506774 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506825 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506872 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506912 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.506962 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507010 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507051 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507085 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507164 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507233 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507278 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507328 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507378 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507415 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 16 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507470 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507523 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507566 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507600 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507646 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507695 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507742 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 16 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507773 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507821 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507863 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507925 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507959 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.507991 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508019 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 16 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508108 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508138 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508188 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508244 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508281 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508325 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 16 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508358 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508420 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508456 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508500 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508544 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508599 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508652 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 16 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508702 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508768 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508823 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508871 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508918 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.508994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 16 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509098 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509153 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509227 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509290 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509341 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509394 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509434 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509507 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509570 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509611 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509662 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509697 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509761 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509844 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509876 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.project_network_types = ['geneve'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509927 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.509981 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510043 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510100 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510148 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510237 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510319 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510367 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510417 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510499 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510553 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510605 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510654 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510691 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510743 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510796 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510839 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510881 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510955 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.510955 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511008 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511034 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511083 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511130 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511165 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511198 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511261 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511299 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511350 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511389 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511443 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511474 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511550 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_rx = 1000 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511584 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_tx = 100 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511621 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_mult = 3 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511676 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511734 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511786 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511822 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511868 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511910 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.511970 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 300 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ha_failover_strategy = normal {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 300 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512433 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512433 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512433 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512433 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512530 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512530 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:199.204.45.129:6641'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512583 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512583 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512636 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512666 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512720 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:199.204.45.129:6642'] {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512753 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512817 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512852 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512902 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.512957 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513017 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513070 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513123 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513181 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513251 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513293 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513334 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513385 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513432 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513485 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513523 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513585 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} May 22 00:57:04.513616 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=105475) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} May 22 00:57:04.514006 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [105475] {{(pid=105475) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} May 22 00:57:04.515378 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-954742', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-26167'] for process (None), after_init {{(pid=105475) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 22 00:57:04.518281 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry May 22 00:57:04.572291 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 22 00:57:04.572291 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 22 00:57:04.572291 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 22 00:57:04.572291 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.572291 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.572694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.572694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.572694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.572694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.572694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.572694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.573051 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.573051 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.573051 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.573051 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.576251 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.129:6641: connecting... May 22 00:57:04.582866 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.129:6641: connected May 22 00:57:04.610797 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry May 22 00:57:04.637470 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 22 00:57:04.637602 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} May 22 00:57:04.637602 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.637602 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=105475) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} May 22 00:57:04.640232 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.129:6642: connecting... May 22 00:57:04.646154 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:199.204.45.129:6642: connected May 22 00:57:04.682990 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683096 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683195 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683279 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683327 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683411 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683643 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-10730'] for port_forwarding_plugin (None), after_init {{(pid=105475) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 22 00:57:04.683700 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683774 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683842 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683903 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.683964 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.684108 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension May 22 00:57:04.684224 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension May 22 00:57:04.684635 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.684720 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685030 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685108 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685155 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685266 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685327 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685371 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685444 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685511 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.685575 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.741711 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.741711 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.741994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.741994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.744059 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.744138 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.744266 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-3952207'] for logging-plugin (None), after_init {{(pid=105475) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 22 00:57:04.744430 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.744503 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:04.744582 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=105475) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} May 22 00:57:14.229543 np0000176101 neutron-ovn-maintenance-worker[104709]: DEBUG dbcounter [-] [104709] Writing DB stats neutron:SELECT=4 {{(pid=104709) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 22 00:57:14.745692 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-05-22 00:57:14.745174 {{(pid=105475) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:99}} May 22 00:57:14.748661 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync port groups started @ 2026-05-22 00:57:14.748105 {{(pid=105475) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:132}} May 22 00:57:14.767351 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG dbcounter [-] [105475] Writer thread running {{(pid=105475) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} May 22 00:57:14.782336 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync port groups completed @ 2026-05-22 00:57:14.781342 {{(pid=105475) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:196}} May 22 00:57:14.782336 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-05-22 00:57:14.781632 {{(pid=105475) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1152}} May 22 00:57:14.893324 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.services.segments.db [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['b016d429-84cd-4312-98ea-7a7d54d0fb86'], 'is_dynamic': False}} {{(pid=105475) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} May 22 00:57:14.897600 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync metadata ports started {{(pid=105475) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1097}} May 22 00:57:15.119148 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync metadata ports completed {{(pid=105475) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1149}} May 22 00:57:15.119343 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=105475) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:949}} May 22 00:57:15.173289 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-05-22 00:57:15.172651 {{(pid=105475) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1022}} May 22 00:57:15.173289 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 22 00:57:15.173522 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=105475) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1027}} May 22 00:57:15.173815 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=105475) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1086}} May 22 00:57:15.173815 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-05-22 00:57:15.173591 {{(pid=105475) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1321}} May 22 00:57:15.173815 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync ACLs started @ 2026-05-22 00:57:15.173687 {{(pid=105475) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:239}} May 22 00:57:15.213998 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync ACLs completed @ 2026-05-22 00:57:15.213769 {{(pid=105475) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:431}} May 22 00:57:15.214128 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync routers and router ports started @ 2026-05-22 00:57:15.214003 {{(pid=105475) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:736}} May 22 00:57:15.267629 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 22 00:57:15.267998 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync routers and router ports completed @ 2026-05-22 00:57:15.267693 {{(pid=105475) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:944}} May 22 00:57:15.268126 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync port QoS policies started @ 2026-05-22 00:57:15.267998 {{(pid=105475) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1369}} May 22 00:57:15.268265 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] Starting OVNClientQosExtension May 22 00:57:15.283393 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync port QoS policies completed @ 2026-05-22 00:57:15.283138 {{(pid=105475) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1392}} May 22 00:57:15.283855 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync Floating IP QoS policies started @ 2026-05-22 00:57:15.283327 {{(pid=105475) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1397}} May 22 00:57:15.283855 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] Starting OVNClientQosExtension May 22 00:57:15.303820 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-05-22 00:57:15.303601 {{(pid=105475) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1420}} May 22 00:57:15.303961 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync Floating IP NAT rules started @ 2026-05-22 00:57:15.303796 {{(pid=105475) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1425}} May 22 00:57:15.304131 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-05-22 00:57:15.304004 {{(pid=105475) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1451}} May 22 00:57:15.304313 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-0e96c746-e3f7-4552-bff0-31b895778388 None None] OVN-Northbound DB sync process completed @ 2026-05-22 00:57:15.304096 {{(pid=105475) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:112}} May 22 00:57:24.747135 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-05-22 00:57:24.746769 {{(pid=105475) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1480}} May 22 00:57:24.748065 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension May 22 00:57:24.749809 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] OVN-SB Sync hostname and physical networks started @ 2026-05-22 00:57:24.749523 {{(pid=105475) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1511}} May 22 00:57:24.754120 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.755180 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.755370 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_dhcp_options_consistency {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.755549 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.756573 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.756659 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.756740 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.756814 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.756889 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.756972 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757045 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757135 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757243 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757574 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757645 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757734 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757809 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.757903 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.758073 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.759144 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] Agents populated in the agent cache: {{(pid=105475) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:309}} May 22 00:57:24.759375 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_ha_failover {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.759561 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] - Agent type: OVN Controller Gateway agent, hostname: np0000176101, name: bf07fdcc-f224-42cb-9bc9-b8ed2e85f3ea {{(pid=105475) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} May 22 00:57:24.759751 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.759928 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] - Agent type: OVN Neutron agent, hostname: np0000176101, name: bf07fdcc-f224-42cb-9bc9-b8ed2e85f3ea {{(pid=105475) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} May 22 00:57:24.761290 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.761390 np0000176101 neutron-ovn-maintenance-worker[105475]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] New host np0000176101 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron May 22 00:57:24.761650 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.761735 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_static_routes {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.762039 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_security_group_with_address_group {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.762130 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_virtual_port_parent_hostname {{(pid=105475) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} May 22 00:57:24.764515 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.765120 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started May 22 00:57:24.769092 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.770320 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization May 22 00:57:24.771718 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.771977 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.772226 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.772502 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.772698 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773175 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773333 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773411 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773488 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773553 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773632 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773738 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773808 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.773947 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774093 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774300 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774387 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774458 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774555 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774630 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774769 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774873 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.774991 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.775118 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' {{(pid=105475) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} May 22 00:57:24.788413 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG oslo_concurrency.lockutils [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=105475) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} May 22 00:57:24.789346 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG oslo_concurrency.lockutils [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.000s {{(pid=105475) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} May 22 00:57:24.805335 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:24.806709 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_baremetal_ports_dhcp_options {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:24.806803 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Sync baremetal port DHCP options with configuration. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:24.809326 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron_lib.callbacks.manager [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-1127439', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-553106'] for segment_host_mapping (None), after_create {{(pid=105475) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} May 22 00:57:24.844377 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.services.segments.db [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] Segments {'0355d723-f329-45fd-b9c7-3c8df319e194'} mapped to the host np0000176101 {{(pid=105475) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} May 22 00:57:24.858237 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG oslo_concurrency.lockutils [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.069s {{(pid=105475) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} May 22 00:57:24.858606 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] OVN-SB Sync hostname and physical networks completed @ 2026-05-22 00:57:24.858236 {{(pid=105475) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1538}} May 22 00:57:24.891066 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_baremetal_ports_dhcp_options finished in 0.084 seconds May 22 00:57:24.891491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:24.891777 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_dhcp_options_consistency {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:24.891943 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Backfill network_id on DHCP_Options external_ids. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:24.946451 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_dhcp_options_consistency finished in 0.054 seconds May 22 00:57:24.946655 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:24.946829 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fair_meter_consistency {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:24.946931 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Refresh ACL logging fair meter after configuration reload. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:24.947284 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fair_meter_consistency finished in 0.000 seconds May 22 00:57:24.947419 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:24.947573 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:24.947675 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:24.949389 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:25.061188 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:25.061695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.068614 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.121 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:25.088583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.089461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:25.090099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.090099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:25.097442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:25.098346 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_for_igmp_snoop_support {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Check for IGMP snoop support. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_for_igmp_snoop_support finished in 0.001 seconds May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.099334 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.099811 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.126807 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Maintenance task: No inconsistencies found. Skipping {{(pid=105475) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:437}} May 22 00:57:25.126963 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_for_inconsistencies finished in 0.028 seconds May 22 00:57:25.127349 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_global_dhcp_opts {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.127466 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Check global DHCP options consistency. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.127720 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_global_dhcp_opts finished in 0.000 seconds May 22 00:57:25.127885 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.128050 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_localnet_port_has_learn_fdb {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.128186 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Ensure localnet ports have learn-fdb set per configuration. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.130410 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 22 00:57:25.132419 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_localnet_port_has_learn_fdb finished in 0.004 seconds May 22 00:57:25.132631 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.132824 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_network_broadcast_arps_to_all_routers {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.132943 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Sync broadcast-arps-to-all-routers on external networks. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.184639 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_network_broadcast_arps_to_all_routers finished in 0.051 seconds May 22 00:57:25.184804 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.184974 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_provider_distributed_ports {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.185067 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Align reside-on-redirect-chassis on VLAN/FLAT distributed ports. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.266908 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_provider_distributed_ports finished in 0.082 seconds May 22 00:57:25.267110 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.267183 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_redirect_type_router_gateway_ports {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.267317 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Align router gateway redirect-type for provider VLAN/FLAT networks. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.353889 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 22 00:57:25.354295 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-f30b4107-3e77-4394-9d5b-d51e904cc1dc None None] OVN-Southbound DB sync process completed @ 2026-05-22 00:57:25.353960 {{(pid=105475) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1488}} May 22 00:57:25.372481 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_redirect_type_router_gateway_ports finished in 0.105 seconds May 22 00:57:25.372634 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.372794 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.372875 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.374151 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:25.374505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.375820 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:25.376847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.377438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:25.377997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:25.378684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: cleanup_old_hash_ring_nodes {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.379529 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Purge stale hash ring nodes older than five days. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.382488 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.db.ovn_hash_ring_db [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Cleaned up Hash Ring nodes older than 5 days May 22 00:57:25.382598 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task cleanup_old_hash_ring_nodes finished in 0.004 seconds May 22 00:57:25.382799 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: configure_nb_global {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.382881 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply ovn_nb_global settings to NB_Global options. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.383152 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:25.383661 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:25.384097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.384907 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:25.385855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.386368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task configure_nb_global failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.386853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:25.387173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:25.387586 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.387586 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:25.387586 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.387586 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:25.387858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.388331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:25.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:25.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:25.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:25.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} May 22 00:57:25.389023 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.389348 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.389564 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.390000 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:57:25.390385 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:25.390584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.391357 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:25.392694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.393107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.394156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:25.394550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:25.394925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.394925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:25.394925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.394925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:25.395909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.396375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_network_type_and_physnet {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.396885 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set network type and physnet external_ids on logical switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.401806 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_network_type_and_physnet finished in 0.005 seconds May 22 00:57:25.401897 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.402012 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_ovn_owned_dns_option {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply ovn_owned option on DNS records per configuration. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.402275 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_ovn_owned_dns_option finished in 0.000 seconds May 22 00:57:25.402360 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.402460 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.402536 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.402688 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:25.403195 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:25.403693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.404704 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:25.406716 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:25.407180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:25.407628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.407989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.408336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:25.408830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:25.409363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.409363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:25.409363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.409363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:25.409363 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_lrouter_ports_ext_ids_name_prefix {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.409363 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: ('Check missing prefix router_name', 'in external_ids of LRPs.') {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.409785 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.001 seconds May 22 00:57:25.409785 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.409785 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.410049 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.410049 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:25.410665 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:25.411076 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:25.413075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.413864 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:25.414691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.414949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:25.415194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.415194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:25.415670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:25.415966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:25.416319 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:25.416319 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:25.416319 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:25.416319 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_router_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.416319 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Sync router.distributed with distributed floating IP config. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.420545 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_router_distributed_flag finished in 0.005 seconds May 22 00:57:25.420700 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.420813 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_router_static_routes {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.420893 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Mark Neutron-owned static routes in OVN external_ids. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.421594 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_router_static_routes finished in 0.001 seconds May 22 00:57:25.421665 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.421796 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_security_group_with_address_group {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:25.421892 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Repair Address_Sets and ACLs for address group rules. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:25.430895 np0000176101 neutron-ovn-maintenance-worker[105475]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_security_group_with_address_group finished in 0.009 seconds May 22 00:57:25.431005 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:25.431673 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=105475) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} May 22 00:57:29.947856 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:29.948607 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:29.948607 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:29.949266 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:29.988244 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:29.988684 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:29.989574 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:29.990371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:29.990947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:29.991571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:29.992276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:29.993154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:29.993154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:29.993154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:29.993154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:29.993154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:29.993154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:30.373294 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:30.375667 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:30.375759 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:30.376689 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:30.377049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.377928 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:30.378931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.379535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:30.380096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:30.380746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.381628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:30.381628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:30.381628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.381628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:30.381628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.381628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:30.389527 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:30.389896 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:30.389982 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:30.390072 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:57:30.390562 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:30.390875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.391593 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:30.392133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.392726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.393123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:30.393700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:30.394275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.394878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.395692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:30.402731 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:30.403054 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:30.403169 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:30.403324 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:30.403964 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:30.404360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.405116 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:30.406383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.407188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.407676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:30.408160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:30.408939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.408939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:30.408939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.408939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.409339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:30.410111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:30.410898 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:30.411538 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:30.411978 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:30.412242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.413035 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:30.413886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.414722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:30.415367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:30.416071 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:30.416990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:30.416990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:30.416990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:30.416990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:30.416990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:30.416990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:34.948587 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:34.949306 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:34.949306 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:34.950096 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:34.984458 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:34.984846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:34.985732 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:34.986467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:34.987081 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:34.987677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:34.988398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:34.989241 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:34.989241 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:34.989241 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:34.989241 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:34.989241 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:34.989241 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:35.378584 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:35.383212 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:35.383212 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:35.384642 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:35.384829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.385431 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:35.386343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.386668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:35.386940 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.386940 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:35.387307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:35.387648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.388031 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:35.388031 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.388031 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:35.389948 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:35.393419 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:35.393509 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:35.393612 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:57:35.393862 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:35.394025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.394414 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:35.395080 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.397407 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:35.397670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:35.397952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.397952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:35.397952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.397952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:35.398323 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.398658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:35.399015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.399015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:35.399015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:35.399015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.399015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:35.399015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.399015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:35.403072 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:35.406317 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:35.406317 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:35.406317 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:35.406317 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:35.406317 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.408362 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:35.409014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.409466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.409842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:35.410077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:35.410454 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.410454 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:35.410454 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.410454 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.412217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:35.412578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:35.412954 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:35.413160 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:35.413160 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:35.413325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.414244 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:35.414993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.415244 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:35.415511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.415511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:35.415863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:35.416179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:35.416575 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:35.416575 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:35.416575 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:39.949171 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:39.949917 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:39.949917 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:39.951016 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:39.984188 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:39.984457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:39.985123 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:39.985889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:39.986761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:39.987360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:39.988168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:39.989061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:39.989061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:39.989061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:39.989061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:39.989061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:39.989061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:40.381814 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:40.382435 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:40.382623 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:40.383473 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:40.383771 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.384530 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:40.386022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.386780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:40.387328 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.387328 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:40.387740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:40.388428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.389257 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:40.389257 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.389257 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:40.393605 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:40.393887 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:40.393993 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:40.394113 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:57:40.394489 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:40.394681 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.395191 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:40.396116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.396589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.397021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:40.397413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:40.398018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.398018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:40.398018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.398018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:40.398149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.398755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:40.399457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.399457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:40.399457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:40.399457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.399457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:40.399457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.399457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:40.405316 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:40.405597 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:40.405757 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:40.405964 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:40.406326 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:40.406555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.407128 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:40.408273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.408753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.409752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:40.410408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:40.411108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.411108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:40.411108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.411108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.411883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:40.412400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:40.413220 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.413220 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:40.413220 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.413220 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:40.413220 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:40.413220 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:40.413220 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:40.413734 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:40.414166 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:40.414626 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:40.414836 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.415502 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:40.416367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.416627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:40.417055 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.417055 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:40.417399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:40.417728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:40.418353 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:40.418353 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:40.418353 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:44.949814 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:44.950542 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:44.950673 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:44.951889 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:44.982926 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:44.983440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:44.984384 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:44.985658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:44.986272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:44.986849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:44.987588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:44.988445 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:44.988445 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:44.988445 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:44.988445 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:44.988445 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:44.988445 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:45.382603 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:45.383281 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:45.383482 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:45.384473 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:45.384796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.385669 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:45.387132 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.387430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:45.387677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.387677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:45.388485 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:45.388817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.389161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:45.389161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.389161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:45.394019 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:45.394349 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:45.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:45.394571 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:57:45.394989 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:45.395348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.396051 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:45.397186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.397480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.398617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:45.398873 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:45.399134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.399134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:45.399134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.399134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:45.400217 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.400594 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:45.401070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.401070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:45.401070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:45.401070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.401070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:45.401070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.401070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:45.405735 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:45.406221 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:45.406352 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:45.406451 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:45.406907 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:45.407314 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.408029 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:45.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.409237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.409512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:45.409834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:45.410252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:45.410616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.411033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:45.413081 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:45.413436 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:45.413598 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:45.413803 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:45.414224 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:45.414572 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:45.414746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.415490 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:45.416436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.416682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:45.416947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.416947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:45.417499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:45.417832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:45.418282 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:45.418282 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:45.418282 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:49.950745 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:49.951401 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:49.951561 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:49.952732 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:49.981601 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:49.981742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:49.982184 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:49.982993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:49.983818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:49.984544 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:49.985226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:49.986040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:49.986040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:49.986040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:49.986040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:49.986040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:49.986040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:50.383387 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:50.383989 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:50.384358 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:50.385031 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:50.385349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.386099 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:50.387482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.387945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:50.388328 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.388328 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:50.389065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:50.389538 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.390036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:50.390036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.390036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:50.394526 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:50.394815 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:50.394929 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:50.395049 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:57:50.395313 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:50.395489 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.395919 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:50.396643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.397075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.397548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:50.397880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:50.398240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.398240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:50.398240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.398240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:50.398923 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.399350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:50.399820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.399820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:50.399820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:50.399820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.399820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:50.399820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.399820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:50.406197 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:50.406581 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:50.406769 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:50.406965 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:50.407363 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:50.407706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.408413 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:50.409596 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.410036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.410981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:50.411348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:50.411733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.411733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:50.411733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.411733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.412732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:50.413263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:50.413783 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.413783 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:50.413783 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.413783 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:50.413783 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:50.413945 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:50.414138 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:50.414384 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:50.414742 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:50.415082 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:50.415266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.415944 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:50.417223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.417553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:50.417887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.417887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:50.418377 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:50.418777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:50.419218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:50.419218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:50.419218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:54.951581 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:54.952531 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:54.952531 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:54.954326 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:54.984705 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:54.984863 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:54.985320 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:54.985960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:54.986543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:54.987193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:54.987936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:54.988990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:54.988990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:54.988990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:54.988990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:54.988990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:54.988990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:55.384005 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:55.384660 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:55.384940 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:55.387767 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:55.388082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.388836 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:55.389693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.390063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:55.390343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.390343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:55.390482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:55.390802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.391157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:55.391157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.391157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:55.394916 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:55.396627 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:55.396712 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:55.396806 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:57:55.397598 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:55.397825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.398385 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:55.399032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.399391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.399655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:55.399907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:55.400184 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.400184 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:55.400184 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.400184 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:55.400427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.400761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:55.401109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.401109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:55.401109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:55.401109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.401109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:55.401109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.401109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:55.406620 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:55.406913 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:55.407069 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:55.407261 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:55.408890 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:55.409402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.410297 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:57:55.411903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.412945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.413971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:55.414867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:55.415843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.415843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:55.415843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.415843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:55.415843 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.417228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:55.418023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:55.418650 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:55.418922 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:55.419014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:55.420397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.420785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:55.421181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.421181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:55.421409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:55.421780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:55.422157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:55.422157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:55.422157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:57:59.952129 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:57:59.952606 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:57:59.952677 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:57:59.953262 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:57:59.980937 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:57:59.980937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:59.981755 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:57:59.982599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:59.983037 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:57:59.983809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:57:59.984571 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:57:59.985432 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:57:59.985432 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:57:59.985432 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:57:59.985432 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:57:59.985432 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:57:59.985432 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:00.384647 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:00.385276 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:00.385475 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.390030 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:00.395419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.397674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:00.399107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:00.400935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:00.403009 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:00.404525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.405572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:00.406612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:00.408235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.410073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.411799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:00.413009 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:00.414255 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.416523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.418547 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.419091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:00.419665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:00.420070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.420652 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:00.421292 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:00.421672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.422116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.422521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:00.423259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:00.423690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:00.424178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:00.424178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:00.424178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:04.952902 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:04.953635 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:04.953635 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:04.954962 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:04.986463 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:04.986890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:04.987736 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:04.988474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:04.989194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:04.989875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:04.990620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:04.991541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:04.991541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:04.991541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:04.991541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:04.991541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:04.991541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:05.385681 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:05.386286 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:05.386537 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:05.387534 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:05.387960 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.388858 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:05.390530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.391194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:05.391840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.391840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:05.392298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:05.393084 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.394088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:05.394088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.394088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:05.397361 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:05.397715 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:05.397910 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:05.398091 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:05.398499 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:05.398777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.399358 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:05.400062 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.400577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.400862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:05.401258 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:05.401665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.401665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:05.401665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.401665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:05.401901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.402411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:05.402945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.402945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:05.402945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:05.402945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.402945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:05.402945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.402945 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:05.407841 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:05.408110 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:05.408217 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:05.408370 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:05.408728 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:05.408992 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.409508 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:05.410271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.410932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.411375 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:05.411910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:05.412692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:05.413390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.414290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:05.417970 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:05.418227 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:05.418364 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:05.418583 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:05.419051 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:05.419441 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:05.419629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.420391 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:05.421674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.422161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:05.422549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.422549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:05.423218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:05.423727 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:05.424337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:05.424337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:05.424337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:09.953659 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:09.954499 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:09.954685 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:09.955944 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:09.992676 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:09.993029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:09.994362 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:09.995007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:09.995574 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:09.996089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:09.996802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:09.998958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:09.998958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:09.998958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:09.998958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:09.998958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:09.998958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:10.386498 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:10.387083 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:10.387321 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:10.388178 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:10.388685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.389787 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:10.391320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.392502 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:10.393628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:10.394457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.395392 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:10.395392 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:10.395392 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.395392 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:10.395392 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.395392 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:10.397900 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:10.398264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:10.398264 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:10.398493 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:10.398869 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:10.399158 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.399855 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:10.400797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:10.401743 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.402239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.402722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:10.403070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.403523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:10.404026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.404026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:10.404026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:10.404026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.404026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:10.404026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.404026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:10.408294 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:10.408628 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:10.408743 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:10.408870 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:10.409310 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:10.409651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.410337 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:10.411235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.412103 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.412668 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:10.413301 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:10.414051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:10.414875 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.415531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:10.418412 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:10.418683 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:10.418784 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:10.418945 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:10.419435 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:10.419788 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:10.419970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.420702 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:10.421555 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.421909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:10.422274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.422274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:10.422511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:10.422983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:10.423984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:10.423984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:10.423984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:14.954484 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:14.956341 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:14.956341 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:14.957004 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:14.959121 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG dbcounter [-] [105475] Writing DB stats neutron:SELECT=520,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=105475) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 22 00:58:14.991272 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:14.991683 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:14.992364 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:14.993480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:14.994133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:14.994854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:14.995691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:14.996740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:14.996740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:14.996740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:14.996740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:14.996740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:14.996740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:15.387369 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:15.388158 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:15.388376 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:15.389473 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:15.389822 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.390789 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:15.392409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.393067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:15.393673 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.393673 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:15.394020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:15.394787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.395763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:15.395763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.395763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:15.398218 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:15.398651 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:15.398763 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:15.398881 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:15.399398 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:15.399703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.400388 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:15.401289 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.401765 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.402036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:15.402466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:15.402864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.402864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:15.402864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.402864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:15.403061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.403560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:15.404065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.404065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:15.404065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:15.404065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.404065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:15.404065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.404065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:15.408610 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:15.408807 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:15.408913 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:15.409036 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:15.409504 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:15.409858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.410543 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:15.411336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.411791 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.412161 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:15.412551 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:15.412943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.412943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:15.412943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.412943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.413180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:15.413745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:15.414630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.414630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:15.414630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.414630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:15.418732 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:15.418873 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:15.419017 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:15.419127 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:15.419629 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:15.420080 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:15.420326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.421086 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:15.422412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.422919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:15.423352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.423352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:15.423949 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:15.424446 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:15.424997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:15.424997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:15.424997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:19.955895 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:19.956439 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:19.956439 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:19.957548 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:19.989025 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:19.989545 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:19.990341 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:19.991045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:19.991577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:19.992616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:19.993481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:19.994798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:19.994798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:19.994798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:19.994798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:19.994798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:19.994798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:20.388179 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:20.388729 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:20.388907 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:20.390019 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:20.390389 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.391300 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:20.392913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.393624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:20.394130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.394130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:20.394729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:20.396429 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.397319 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:20.397319 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.397319 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:20.398630 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:20.398965 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:20.399126 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:20.399327 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:20.400091 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:20.400469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.401300 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:20.402131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.402570 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.402968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:20.403334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:20.403732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.403732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:20.403732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.403732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:20.404086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.404540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:20.405038 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.405038 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:20.405038 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:20.405038 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.405038 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:20.405038 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.405038 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:20.408960 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:20.409295 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:20.409428 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:20.409554 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:20.410030 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:20.410475 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.411221 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:20.412261 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.412702 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.413383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:20.413739 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:20.414129 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.414129 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:20.414129 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.414129 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.414877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:20.415450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:20.416299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.416299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:20.416299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.416299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:20.419038 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:20.419334 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:20.419489 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:20.419674 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:20.420132 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:20.420500 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:20.420695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.421388 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:20.422229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.422761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:20.423111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.423111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:20.423179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:20.423634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:20.424135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:20.424135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:20.424135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:24.956582 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:24.957512 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:24.957512 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:24.958495 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:24.989146 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:24.989560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:24.990449 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:24.991605 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:24.992255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:24.992843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:24.992843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:24.993120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:24.993707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:24.994286 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:24.994286 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:24.994286 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:25.389016 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:25.389560 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:25.389866 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:25.391048 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:25.391434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.392460 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:25.394590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.395516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:25.396576 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.396576 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:25.396922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:25.397491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.398045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:25.398045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.398045 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:25.398918 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:25.399120 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:25.399252 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:25.399368 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:25.399809 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:25.400039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.400682 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:25.401515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.401978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.402352 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:25.402712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:25.403126 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.403126 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:25.403126 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.403126 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:25.403385 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.403844 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:25.404479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.404479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:25.404479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:25.404479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.404479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:25.404479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.404479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:25.409309 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:25.409516 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:25.409647 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:25.409783 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:25.410123 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:25.410365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.410872 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:25.411665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.412121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.412553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:25.412927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:25.413601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:25.414358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.415344 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:25.419721 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:25.419943 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:25.420064 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:25.420279 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:25.420660 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:25.420860 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:25.420927 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.421420 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:25.422270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.422907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:25.423521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:25.424448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:25.425317 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:25.425317 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:25.425317 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:25.425317 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:25.425317 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:25.425317 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:29.957136 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:29.957775 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:29.957949 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:29.959225 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:29.990961 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:29.991410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:29.992249 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:29.993387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:29.993820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:29.994238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:29.994238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:29.994481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:29.995506 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:29.996753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:29.996753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:29.996753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:30.389684 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:30.390339 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:30.390537 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:30.391416 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:30.391694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.392524 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:30.393907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.394674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:30.395417 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:30.396195 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.397185 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:30.397185 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:30.397185 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.397185 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:30.397185 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.397185 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:30.399327 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:30.399596 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:30.399769 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:30.399769 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:30.400316 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:30.400696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.401403 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:30.402077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.402606 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.402910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:30.403299 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:30.403717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.403717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:30.403717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.403717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:30.403843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.404331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:30.404977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.404977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:30.404977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:30.404977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.404977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:30.404977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.404977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:30.409685 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:30.409981 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:30.410096 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:30.410216 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:30.410505 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:30.410721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.411154 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:30.411829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.412348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.412629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:30.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:30.413434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.413434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:30.413434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.413434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.413703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:30.414713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:30.415850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.415850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:30.415850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.415850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:30.420071 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:30.420571 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:30.420977 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:30.421219 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:30.421499 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:30.421720 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:30.421820 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.422335 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:30.423691 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.424249 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:30.424757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.424757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:30.425339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:30.425890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:30.426524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:30.426524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:30.426524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:34.957894 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:34.958826 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:34.959016 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:34.960303 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:34.997228 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:34.997561 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:34.998601 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:34.999674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.000326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:35.000878 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:35.001595 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.002685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:35.002685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:35.002685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.002685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:35.002685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.002685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:35.390347 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:35.390679 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:35.390824 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:35.392124 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:35.392659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.393713 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:35.394887 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.395437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:35.395971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:35.396604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.396941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:35.396941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:35.396941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.396941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:35.396941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.396941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:35.399645 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:35.399882 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:35.399993 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:35.400109 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:35.400750 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:35.401089 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.402083 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:35.402869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.403152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.403678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:35.403915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:35.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:35.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:35.404709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.405157 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:35.405640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.405640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:35.405640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:35.405640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.405640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:35.405640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.405640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:35.410047 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:35.410302 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:35.410407 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:35.410540 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:35.411080 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:35.411467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.412193 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:35.412991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.413440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.414194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:35.414535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:35.414902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.414902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:35.414902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.414902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.415922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:35.416409 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:35.416954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.416954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:35.416954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.416954 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:35.420643 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:35.421021 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:35.421185 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:35.421366 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:35.421750 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:35.421985 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:35.422112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.422695 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:35.423582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.423950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:35.424330 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.424330 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:35.424676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:35.425807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:35.426530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:35.426530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:35.426530 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:39.958890 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:39.959330 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:39.959547 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:39.960789 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:40.010490 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:40.010984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.012325 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:40.013688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.014381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:40.014976 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:40.015704 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.016579 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:40.016579 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:40.016579 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.016579 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:40.016579 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.016579 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:40.390895 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:40.391482 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:40.391680 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:40.393006 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:40.393458 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.394649 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:40.396382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.397004 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:40.397614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.397614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:40.398106 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:40.398849 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.399722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:40.399722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.399722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:40.400086 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:40.400452 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:40.400627 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:40.400805 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:40.401306 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:40.401560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.402184 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:40.402980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:40.403757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.404675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.405373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:40.406127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.406660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:40.407190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.407190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:40.407190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:40.407190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.407190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:40.407190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.407190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:40.410505 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:40.410912 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:40.411038 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:40.411310 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:40.411863 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:40.412247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.413040 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:40.414357 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.415135 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.415809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:40.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:40.417149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.417149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:40.417149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.417149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.417817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:40.418862 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:40.419880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.419880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:40.419880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.419880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:40.421101 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:40.421359 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:40.421448 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:40.421605 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:40.422119 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:40.422525 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:40.422710 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.423600 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:40.424607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.425056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:40.425467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.425467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:40.425644 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:40.426190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:40.426766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:40.426766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:40.426766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:44.959648 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:44.960615 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:44.960615 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:44.961694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:44.997187 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:44.997564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:44.998696 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:44.999732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.000356 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:45.000882 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:45.001616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.002477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:45.002477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:45.002477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.002477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:45.002477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.002477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:45.391673 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:45.392346 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:45.392529 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:45.393944 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:45.394481 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.395560 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:45.397232 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.398835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:45.400332 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:45.401663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:45.402694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.402987 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:45.403733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:45.404758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.405550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.406290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:45.406636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.407044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:45.407516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.407516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:45.407516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:45.407516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.407516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:45.407516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.407516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:45.410979 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:45.411291 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:45.411385 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:45.411512 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:45.412038 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:45.412447 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.413187 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:45.413934 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:45.414840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:45.415403 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:45.415779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:45.416092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.416599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:45.421528 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:45.421835 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:45.421935 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:45.422087 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:45.422614 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:45.422945 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:45.423124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.423883 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:45.424776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.425108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:45.425440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.425440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:45.425709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:45.426120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:45.426932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:45.426932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:45.426932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:49.960253 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:49.960894 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:49.961116 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:49.962588 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:49.997345 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:49.997715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:49.998559 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:49.999335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.000281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:50.000647 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:50.001077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.001631 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:50.001631 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:50.001631 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.001631 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:50.001631 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.001631 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:50.392425 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:50.392976 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:50.393165 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:50.395095 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:50.395592 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.396730 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:50.398379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.398942 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:50.399540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.399540 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:50.400090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:50.400826 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.401699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:50.401699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.401699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:50.401699 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:50.401699 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:50.401699 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:50.401699 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:50.402168 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:50.402440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.403163 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:50.404423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.405072 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.405871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:50.406449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:50.407152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.407152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:50.407152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.407152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:50.407553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.408350 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:50.409437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.409437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:50.409437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:50.409437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.409437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:50.409437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.409437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:50.411468 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:50.411761 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:50.411872 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:50.411987 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:50.412600 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:50.412975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.413776 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:50.414582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.414997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.416011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:50.416358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:50.416752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.416752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:50.416752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.416752 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.417814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:50.418351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:50.419009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.419009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:50.419009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.419009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:50.422227 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:50.422623 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:50.422730 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:50.422869 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:50.423345 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:50.423589 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:50.423728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.424499 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:50.425370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.425703 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:50.426056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.426056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:50.426348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:50.426991 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:50.428228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:50.428228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:50.428228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:54.961070 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:54.961839 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:54.962033 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:54.963372 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:54.995994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:54.996599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:54.997850 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:54.998903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:54.999463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:54.999978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:55.000625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.001437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:55.001437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:55.001437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.001437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:55.001437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.001437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:55.393070 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:55.393640 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:55.393870 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:55.394688 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:55.394977 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.395801 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:55.397248 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.398556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:55.399627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:55.400256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.401010 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:55.401010 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:55.401010 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.401010 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:55.401010 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.401010 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:55.401993 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:55.401993 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:55.401993 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:55.401993 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:58:55.402393 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:55.402622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.403416 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:55.404792 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.405663 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.406292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:55.406790 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:55.407144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.407144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:55.407144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.407144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:55.407610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.408494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:55.409228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.409228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:55.409228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:55.409228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.409228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:55.409228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.409228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:55.411896 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:55.412151 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:55.412283 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:55.412409 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:55.412676 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:55.413025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.413711 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:58:55.414478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.414906 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.415487 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:55.416099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:55.416859 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:55.417236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.417760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:55.422708 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:55.422987 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:55.423084 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:55.423255 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:58:55.423641 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:55.423998 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:58:55.424219 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.424890 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:58:55.425675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.426026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:58:55.426362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.426362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:58:55.427390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:58:55.428131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:58:55.428920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:58:55.428920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:58:55.428920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:58:59.962089 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:58:59.963329 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:58:59.963508 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:58:59.965274 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:00.001028 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:00.001586 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.002744 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:00.003835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.005156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:00.005858 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:00.007565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.009164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:00.009164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:00.009164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.009164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:00.009164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.009164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:00.393692 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:00.394360 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:00.394552 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:00.395658 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:00.396031 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.397055 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:00.398590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.399238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:00.399814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.399814 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:00.400200 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:00.401023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.401928 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:00.401928 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.401928 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:00.401928 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:00.402133 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:00.402133 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:00.402603 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:00.402758 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:00.403128 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.403857 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:00.404604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.405268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.405564 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:00.405947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:00.406364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.406364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:00.406364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.406364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:00.406534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.406984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:00.407523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.407523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:00.407523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:00.407523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.407523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:00.407523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.407523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:00.412686 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:00.413100 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:00.413282 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:00.413457 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:00.413847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:00.414112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.414726 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:00.415909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.416387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.417439 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:00.418065 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:00.418688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.418688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:00.418688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.418688 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.419230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:00.420133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:00.421128 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.421128 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:00.421128 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.421128 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:00.423139 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:00.423381 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:00.423472 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:00.423631 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:00.424031 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:00.424303 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:00.424436 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.424971 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:00.425837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.426250 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:00.426850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:00.427341 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:00.427955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:00.427955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:00.427955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:00.427955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:00.427955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:00.427955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:04.962876 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:04.963612 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:04.963612 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:04.964940 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:04.999670 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:05.000268 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.000628 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:05.001669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.002335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:05.003033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:05.004110 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.005155 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:05.005155 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:05.005155 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.005155 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:05.005155 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.005155 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:05.394747 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:05.395332 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:05.395635 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:05.396905 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:05.397370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.398514 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:05.400442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.400837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:05.401276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.401276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:05.401896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:05.402363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.402916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:05.402916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.402916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:05.402916 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:05.402916 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:05.402916 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:05.402916 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:05.403320 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:05.403548 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.404181 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:05.405008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:05.405833 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.406391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.406831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:05.407247 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:05.408379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.408379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:05.408379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:05.408379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.408379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:05.408379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.408379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:05.413106 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:05.413380 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:05.413487 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:05.413623 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:05.414050 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:05.414351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.414944 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:05.415722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.416223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.416664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:05.417051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:05.417496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.417496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:05.417496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.417496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.418125 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:05.418648 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:05.419239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.419239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:05.419239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.419239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:05.423453 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:05.423815 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:05.424031 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:05.424378 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:05.424781 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:05.425036 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:05.425168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:05.426613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.427023 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:05.427428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.427428 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:05.427585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:05.428078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:05.428662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:05.428662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:05.428662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:09.963496 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:09.964366 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:09.964366 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:09.965447 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:09.997532 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:09.999078 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:09.999672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.000335 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:10.000994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:10.001796 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.002705 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:10.002705 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:10.002705 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.002705 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:10.002705 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.002705 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:10.395485 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:10.396144 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:10.396395 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:10.397484 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:10.397802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.398685 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:10.400194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.400843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:10.401434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.401434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:10.403469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:10.404240 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.404742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:10.404742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.404742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:10.404742 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:10.404742 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:10.404742 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:10.404742 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:10.405024 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:10.405351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.406101 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:10.407127 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.407601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.407939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:10.408326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:10.408721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.408721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:10.408721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.408721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:10.408922 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.409415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:10.409916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.409916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:10.409916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:10.409916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.409916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:10.409916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.409916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:10.413521 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:10.413814 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:10.413926 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:10.414052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:10.414565 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:10.414917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.415689 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:10.416413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.416831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.417365 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:10.417706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:10.418094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.418094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:10.418094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.418094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.419150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:10.419672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:10.420196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.420196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:10.420196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.420196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:10.423876 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:10.424397 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:10.424567 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:10.424847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:10.425279 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:10.425488 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:10.425600 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.426158 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:10.427076 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.427480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:10.427850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.427850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:10.428091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:10.428562 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:10.429086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:10.429086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:10.429086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:14.964583 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:14.965348 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:14.965348 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:14.966470 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:14.997649 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG dbcounter [-] [105475] Writing DB stats neutron:SELECT=255 {{(pid=105475) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 22 00:59:15.002804 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:15.003528 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.004947 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:15.006063 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.006453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:15.006803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.006803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:15.007107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:15.007563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.008091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:15.008091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.008091 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:15.396184 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:15.396647 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:15.396858 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:15.397693 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:15.398131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.398780 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:15.400309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.400712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:15.401102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.401102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:15.402116 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:15.402853 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.403422 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:15.403422 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.403422 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:15.404131 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:15.404596 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:15.404596 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:15.404596 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:15.405185 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:15.405500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.406183 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:15.407105 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.407733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.408017 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:15.408397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:15.408795 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.408795 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:15.408795 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.408795 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:15.408935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.409421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:15.409965 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.409965 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:15.409965 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:15.409965 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.409965 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:15.409965 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.409965 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:15.414029 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:15.414320 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:15.414463 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:15.414612 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:15.415099 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:15.415431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.416032 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:15.416901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.417470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.418491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:15.418834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:15.419379 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:15.419806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.420396 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:15.424401 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:15.424704 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:15.424844 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:15.424997 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:15.425556 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:15.426035 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:15.426126 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.426834 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:15.427786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.428181 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:15.428617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.428617 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:15.428803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:15.429325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:15.429905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:15.429905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:15.429905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:19.965166 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:19.965688 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:19.965762 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:19.966390 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:19.997996 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:19.998426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:19.999253 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:20.000025 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.000805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:20.001390 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:20.002235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.003070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:20.003070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:20.003070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.003070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:20.003070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.003070 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:20.397757 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:20.398409 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.400389 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:20.401854 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.402274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:20.402714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.402714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:20.403848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:20.404325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.404832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:20.404832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.404832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:20.404832 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:20.405041 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:20.405329 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:20.405540 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:20.405845 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:20.406021 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.406492 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:20.407239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.407659 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.408018 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:20.408450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:20.408932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.408932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:20.408932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.408932 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:20.409050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.409501 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:20.409978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.409978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:20.409978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:20.409978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.409978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:20.409978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.409978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:20.414378 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:20.414741 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:20.414861 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:20.415006 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:20.415644 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:20.415907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.416462 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:20.417450 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.418121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.418554 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:20.419019 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:20.419689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:20.420373 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.421477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:20.424799 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:20.425027 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:20.425124 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:20.425308 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:20.425777 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:20.426182 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:20.426465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.427307 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:20.428100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.428696 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:20.429362 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:20.430059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:20.430950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:20.430950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:20.430950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:20.430950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:20.430950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:20.430950 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:24.966516 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:24.967137 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:24.967382 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:24.968552 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:24.996370 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:24.996534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:24.996973 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:24.997707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:24.998303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:24.998889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:24.999638 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.000179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:25.000179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:25.000179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.000179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:25.000179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.000179 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:25.398483 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:25.399036 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:25.399249 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:25.399979 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:25.400246 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.400992 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:25.403197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.404404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:25.405160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:25.405911 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:25.406847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.407484 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:25.408225 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.408603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.409013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:25.409623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:25.410315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.410789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.411256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:25.414876 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:25.415105 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:25.415183 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:25.415285 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:25.415460 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:25.415624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.415910 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:25.416430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.416695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.418092 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:25.418315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:25.418556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.418556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:25.418556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.418556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.420003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:25.420339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:25.420678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.420678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:25.420678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.420678 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:25.425167 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:25.425470 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:25.425580 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:25.425742 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:25.426034 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:25.426509 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:25.426509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.427231 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:25.427672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.428029 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:25.428380 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.428380 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:25.428709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:25.429252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:25.429957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:25.429957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:25.429957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:29.967288 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:29.967885 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:29.968071 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:29.969076 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:30.002491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:30.002685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.003399 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:30.004371 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.004697 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:30.005002 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.005002 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:30.005525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:30.005961 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.006467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:30.006467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.006467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:30.398935 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:30.399179 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:30.399285 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:30.399717 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:30.399839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.400241 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:30.401069 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.401664 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:30.402327 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:30.402872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.403550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:30.403550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:30.403550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.403550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:30.403550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.403550 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:30.405319 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:30.405495 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:30.405576 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:30.405661 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:30.405892 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:30.406060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.406563 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:30.407326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.407800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.408174 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:30.408549 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:30.408985 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.408985 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:30.408985 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.408985 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:30.411189 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.411939 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:30.412479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.412479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:30.412479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:30.412479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.412479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:30.412479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.412479 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:30.415215 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:30.415431 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:30.415525 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:30.415646 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:30.416577 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:30.417090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.417990 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:30.419421 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.419889 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.421051 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:30.421676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:30.422086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.422086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:30.422086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.422086 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.426798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:30.427279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:30.427740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.427740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:30.427740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.427740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:30.427740 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:30.427740 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:30.429053 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:30.429178 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:30.429459 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:30.429638 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:30.429699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.431221 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:30.431776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.432386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:30.432982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:30.433729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:30.434649 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:30.434649 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:30.434649 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:30.434649 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:30.434649 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:30.434649 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:34.967928 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:34.968730 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:34.968730 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:34.969938 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:34.997764 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:34.997972 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:34.998306 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:34.998926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:34.999589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:35.000731 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:35.001505 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.002486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:35.002486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:35.002486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.002486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:35.002486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.002486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:35.399527 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:35.400166 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:35.400386 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:35.401151 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:35.401467 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.402380 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:35.403800 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.404433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:35.405020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.405020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:35.405535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:35.406224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:35.406837 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:35.407093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.407519 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:35.408139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.408578 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.408956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:35.409347 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:35.409735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.409735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:35.409735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.409735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:35.409941 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.410404 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:35.410905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.410905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:35.410905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:35.410905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.410905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:35.410905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.410905 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:35.415631 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:35.415956 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:35.416065 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:35.416192 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:35.416448 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:35.416660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.417108 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:35.417789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.418271 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.418624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:35.419160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:35.419575 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.419575 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:35.419575 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.419575 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.419699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:35.420355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:35.421060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.421060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:35.421060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.421060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:35.427910 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:35.428222 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:35.428352 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:35.428499 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:35.428710 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:35.428938 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:35.429043 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.429910 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:35.431662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.432139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:35.432933 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.432933 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:35.433386 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:35.434178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:35.435077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:35.435077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:35.435077 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:39.968708 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:39.969648 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:39.969648 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:39.970918 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:40.010093 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:40.010627 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.011789 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:40.012931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.013714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:40.014542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:40.015325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.016064 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:40.016064 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:40.016064 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.016064 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:40.016064 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.016064 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:40.400987 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:40.401254 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:40.401465 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:40.402884 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:40.403453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.404719 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:40.406426 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.406775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:40.407131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.407131 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:40.407131 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:40.408639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:40.409085 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.409636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:40.409636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.409636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:40.409636 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:40.409636 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:40.409636 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:40.409817 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:40.410095 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.410680 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:40.411496 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.411935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.412256 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:40.412590 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:40.412962 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.412962 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:40.412962 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.412962 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:40.413311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.413732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:40.414242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.414242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:40.414242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:40.414242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.414242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:40.414242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.414242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:40.416042 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:40.416390 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:40.416512 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:40.416642 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:40.417031 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:40.417298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.417902 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:40.418794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.419226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.419737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:40.420067 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:40.420519 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.420519 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:40.420519 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.420519 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.420806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:40.421349 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:40.421896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.421896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:40.421896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.421896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:40.428432 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:40.428798 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:40.428914 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:40.429080 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:40.429638 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:40.429989 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:40.430178 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.430936 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:40.431830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.432333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:40.432895 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:40.433553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:40.434359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:40.434359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:40.434359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:40.434359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:40.434359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:40.434359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:44.969915 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:44.970458 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:44.970954 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:44.972234 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:45.002172 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:45.002607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.003263 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:45.003959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.004430 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:45.006100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:45.006518 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.007230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:45.007230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:45.007230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.007230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:45.007230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.007230 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:45.431193 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:45.431193 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:45.431193 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:45.431193 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:45.431844 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:45.432110 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:45.432952 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:45.433263 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.434111 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:45.435624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.436509 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:45.437395 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.437395 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:45.437552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:45.438471 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:45.439747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.440267 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:45.441082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:45.443832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.444419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.444672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:45.444917 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.445266 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:45.445612 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:45.445829 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:45.445829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.446303 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:45.447030 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.447355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.447921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:45.448141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:45.448427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.448427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:45.448427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.448427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.448988 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:45.449337 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:45.449683 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:45.449919 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:45.450082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.450498 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:45.451022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.451290 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:45.451531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.451531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:45.451694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:45.451994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:45.452489 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:45.452489 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:45.452489 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:49.970517 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:49.971494 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:49.971494 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:49.972692 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:50.013169 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:50.013937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.015243 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:50.016462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.017133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:50.018066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.018066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:50.018345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:50.019094 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.019670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:50.019670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.019670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:50.431952 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:50.433491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:50.433491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:50.434391 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:50.434772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.436008 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:50.437973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.438637 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:50.439245 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.439245 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:50.439499 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:50.440196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.440864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:50.440864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.440864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:50.440864 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:50.440864 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:50.440864 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:50.440864 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:50.441359 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:50.441583 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.442190 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:50.443100 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.443580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.445044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:50.445423 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:50.445797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.445797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:50.445797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.445797 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:50.446345 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.446866 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:50.447584 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:50.448042 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:50.448042 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:50.448042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.448830 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:50.450036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:50.450483 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.451367 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:50.451692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:50.452054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.452054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:50.452054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.452054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.452517 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:50.453074 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:50.453847 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:50.454242 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:50.454242 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.454733 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:50.455646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.455994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:50.456397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.456397 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:50.456722 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:50.457162 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:50.458015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:50.458015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:50.458015 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:54.971424 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:54.972335 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:54.972335 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:54.973473 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:55.000794 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:55.000958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.001369 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:55.002098 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.003288 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:55.004238 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:55.004855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.006180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:55.006180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:55.006180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.006180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:55.006180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.006180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:55.432813 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:55.433473 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:55.433660 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:55.434428 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:55.434778 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.435724 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:55.437160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.437748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:55.438267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.438267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:55.438920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:55.439959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.440437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:55.440437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.440437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:55.440611 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:55.441012 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:55.441198 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:55.441391 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 00:59:55.441634 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:55.441807 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.442343 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:55.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.443449 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.443994 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:55.444676 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:55.445281 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.446096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:55.446986 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:55.447260 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:55.447363 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:55.447510 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:55.447765 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:55.447982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.448418 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 00:59:55.449270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:55.450104 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:55.450876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:55.451438 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 00:59:55.451724 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.452167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:55.452852 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:55.453106 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:55.453216 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:55.453381 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 00:59:55.453621 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:55.453825 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 00:59:55.453935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.454381 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 00:59:55.455112 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.455831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 00:59:55.456542 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 00:59:55.456964 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 00:59:55.457441 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 00:59:55.457441 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 00:59:55.457441 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 00:59:55.457441 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 00:59:55.457441 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 00:59:55.457441 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 00:59:59.972118 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 00:59:59.973110 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 00:59:59.973110 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 00:59:59.974231 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:00.014801 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:00.015060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.016130 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:00.018500 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.019272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:00.019817 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:00.020640 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.021451 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:00.021451 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:00.021451 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.021451 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:00.021451 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.021451 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:00.433791 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:00.434304 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:00.434495 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:00.435580 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:00.435901 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.436775 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:00.438634 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.439164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:00.439904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.439904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:00.440602 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:00.442007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:00.442846 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.443259 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:00.443641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:00.445059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.446525 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.447283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:00.447909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.448787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:00.449559 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.449916 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:00.450187 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.450568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.450984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:00.451287 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:00.451620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.451620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:00.451620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.451620 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.452304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:00.452746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:00.453197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.453197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:00.453197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.453197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:00.453197 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:00.453556 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:00.453662 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:00.453885 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:00.454401 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:00.454804 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:00.454982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.455683 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:00.456639 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.456880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:00.457129 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.457129 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:00.457832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:00.458166 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:00.458521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:00.458521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:00.458521 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:04.972976 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:04.973701 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:04.973829 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:04.974848 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:05.282841 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:05.283282 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.284148 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.310 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:05.284952 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.285315 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:05.285670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.285670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:05.285751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:05.286171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.286694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:05.286694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.286694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:05.434443 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:05.434949 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:05.435105 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:05.436270 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:05.436665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.437587 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:05.439093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.439677 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:05.440513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.440513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:05.440794 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:05.442326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:05.443052 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:05.443483 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.444034 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:05.444848 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:05.445840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.446498 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.447348 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:05.448270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.448811 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:05.449695 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.450320 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:05.450798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.451167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.451622 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:05.451913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:05.452801 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:05.453363 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:05.453904 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:05.454143 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:05.454143 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:05.454565 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:05.454932 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:05.455118 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.455788 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:05.456672 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.457226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:05.457654 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:05.458102 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:05.458495 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:05.458495 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:05.458495 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:05.458495 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:05.458495 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:05.458495 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:09.973690 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:09.974487 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:09.974487 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:09.976178 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:10.004369 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:10.004757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.005710 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:10.007090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.007798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:10.008387 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:10.009144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.010411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:10.010411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:10.010411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.010411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:10.010411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.010411 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:10.435250 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:10.435922 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:10.436129 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:10.437048 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:10.437419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.438286 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:10.439931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.440463 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:10.441014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.441014 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:10.441657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:10.442306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:10.443891 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:10.444255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:10.445580 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.446253 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.446978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:10.447304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:10.447643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.447643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:10.447643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.447643 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:10.448325 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.448730 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:10.449148 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:10.449416 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:10.449416 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:10.449870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.450096 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:10.450805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:10.451746 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:10.452334 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.452636 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.452902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:10.453326 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:10.453771 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.453771 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:10.453771 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.453771 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:10.453874 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:10.454531 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:10.454531 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:10.454531 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:10.454759 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:10.455025 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:10.455212 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.455938 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:10.456740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.457298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:10.457869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:10.458529 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:10.459022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:10.459022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:10.459022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:10.459022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:10.459022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:10.459022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:14.974494 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:14.978298 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:14.978298 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:14.978298 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:15.004686 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG dbcounter [-] [105475] Writing DB stats neutron:SELECT=244 {{(pid=105475) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 22 01:00:15.006605 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:15.006778 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.007226 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:15.009434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.010144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:15.010729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:15.011431 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.012280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:15.012280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:15.012280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.012280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:15.012280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.012280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:15.435996 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:15.436671 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:15.436867 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:15.437756 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:15.438020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.438840 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:15.440218 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.440819 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:15.441384 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.441384 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:15.441974 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:15.442747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:15.443582 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.443968 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:15.445512 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:15.447035 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.448255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.448926 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:15.449511 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.449903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:15.450360 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.450623 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:15.451226 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:15.452420 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:15.453154 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:15.453938 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:15.454394 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.455308 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:15.455755 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:15.455755 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:15.455755 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:15.455755 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:15.455755 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:15.455755 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:15.455886 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.456137 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:15.456913 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.457419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:15.457665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.457665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:15.457978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:15.458284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:15.458614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:15.458614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:15.458614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:19.975925 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:19.976925 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:19.977127 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:19.978476 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:20.011877 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:20.011877 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.012229 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:20.013097 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.013462 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:20.013786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.013786 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:20.014139 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:20.014577 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.015043 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:20.015043 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.015043 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:20.436808 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:20.438937 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:20.438937 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:20.439835 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:20.440304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:20.443022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:20.443556 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.443935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:20.443935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:20.443935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.443935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:20.443935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.443935 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:20.445243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:20.445715 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.446272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:20.446272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.446272 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:20.446570 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:20.447263 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:20.447263 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:20.447263 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:20.447773 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:20.448093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.450027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.450619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.451669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:20.453675 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:20.453983 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:20.454304 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:20.454729 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:20.455111 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:20.455111 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:20.455111 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:20.455111 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.455111 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.457850 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:20.458239 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:20.458537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.458789 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:20.459044 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:20.459437 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:20.459802 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:20.460027 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:20.460027 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:20.460027 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.460311 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:20.461117 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.461413 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:20.461660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.461660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:20.462434 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:20.462748 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:20.463087 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:20.463087 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:20.463087 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:24.977056 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:24.977815 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:24.977815 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:24.979077 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:25.005817 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:25.006235 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.007197 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:25.009121 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.009740 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:25.010351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.010351 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:25.011374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:25.012267 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.013153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:25.013153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.013153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:25.437790 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:25.438329 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:25.438529 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:25.439688 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:25.440012 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.440903 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:25.442557 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.443243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:25.443799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.443799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:25.444254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:25.444979 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.445772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:25.445772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.445772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:25.447116 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:25.447414 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:25.447538 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:25.447647 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:25.448148 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:25.448466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.449159 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:25.449956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.450472 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.450981 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:25.451388 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:25.451787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.451787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:25.451787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.451787 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:25.452046 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.452523 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:25.453050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.453050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:25.453050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:25.453050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.453050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:25.453050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.453050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:25.453341 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:25.453639 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:25.453748 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:25.453870 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:25.454388 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:25.454776 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.455475 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:25.456279 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.456732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.457134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:25.457503 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:25.457879 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.457879 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:25.457879 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.457879 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.458329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:25.458912 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:25.459535 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:25.459814 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:25.459855 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:25.460054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.460747 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:25.461565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.461909 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:25.462269 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.462269 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:25.462569 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:25.463049 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:25.463581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:25.463581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:25.463581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:29.978018 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:29.978948 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:29.978948 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:29.980238 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:30.012918 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:30.013368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.014141 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:30.014884 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.015709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:30.016515 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:30.017486 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.018646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:30.018646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:30.018646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.018646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:30.018646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.018646 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:30.438527 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:30.439078 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:30.439341 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:30.440477 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:30.440831 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.441751 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:30.443306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.443868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:30.444440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.444440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:30.444925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:30.445655 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.446455 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:30.446455 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.446455 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:30.447598 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:30.447999 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:30.448164 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:30.448369 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:30.448854 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:30.449137 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.450011 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:30.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.451723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.452690 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:30.453040 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:30.453469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.453469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:30.453469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.453469 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:30.454492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.454918 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:30.455398 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:30.455644 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:30.455644 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:30.456127 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:30.456516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.457402 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:30.458761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:30.459168 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.460050 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:30.460803 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:30.461173 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:30.461604 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:30.462124 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:30.462491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:30.462491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:30.462491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:30.462491 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.462806 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:30.463400 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.463742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:30.464061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:30.464478 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:30.464989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:30.464989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:30.464989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:30.464989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:30.464989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:30.464989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:34.978904 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:34.979718 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:34.979836 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:34.981274 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:35.007385 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:35.007763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.008497 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:35.009159 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.009766 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:35.010453 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:35.011078 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.012009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:35.012009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:35.012009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.012009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:35.012009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.012009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:35.439168 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:35.439753 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:35.439897 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:35.440846 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:35.441172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.441968 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:35.442982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.443584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:35.444180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:35.444871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.445693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:35.445693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:35.445693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.445693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:35.445693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.445693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:35.448162 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:35.448502 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:35.448619 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:35.448689 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:35.449135 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:35.449457 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.450088 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:35.450813 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.451302 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.451568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:35.451919 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:35.452339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.452339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:35.452339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.452339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:35.452613 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.453022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:35.453563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.453563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:35.453563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:35.453563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.453563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:35.453563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.453563 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:35.455323 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:35.455631 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:35.455748 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:35.455913 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:35.456346 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:35.456700 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.457391 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:35.458109 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.458514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.458955 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:35.459300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:35.459692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.459692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:35.459692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.459692 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.460026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:35.460470 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:35.460966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.460966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:35.460966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.460966 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:35.461083 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:35.461395 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:35.461511 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:35.461657 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:35.462061 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:35.462425 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:35.462609 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.463295 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:35.464120 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.464465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:35.464835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.464835 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:35.465066 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:35.465514 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:35.466000 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:35.466000 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:35.466000 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:39.979684 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:39.980357 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:39.980599 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:39.981845 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:40.014685 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:40.015000 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.015679 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:40.016657 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.017022 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:40.017440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.017440 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:40.017689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:40.018130 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.018624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:40.018624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.018624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:40.440078 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:40.440730 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:40.440928 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:40.442902 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:40.443149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.443984 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:40.445513 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.446082 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:40.446660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.446660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:40.447114 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:40.447846 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.448674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:40.448674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.448674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:40.448674 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:40.449057 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:40.449235 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:40.449457 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:40.449912 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:40.450243 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.451048 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:40.452300 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.452742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.453682 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:40.454060 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:40.454466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.454466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:40.454466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.454466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:40.455408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.455876 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:40.456410 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:40.456688 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:40.457759 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:40.458149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.458962 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:40.459687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.460101 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.460565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:40.460903 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:40.461297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.461297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:40.461297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.461297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.461599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:40.462068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:40.462568 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:40.462938 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:40.463339 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:40.463534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.464009 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:40.464864 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.465236 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:40.465610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.465610 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:40.465805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:40.466273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:40.466781 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:40.466781 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:40.466781 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:44.980501 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:44.981354 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:44.981565 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:44.982793 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:45.022751 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:45.023149 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.024003 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:45.024774 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.025424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:45.025958 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:45.026616 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.027552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:45.027552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:45.027552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.027552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:45.027552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.027552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:45.440697 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:45.441472 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:45.441706 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:45.442567 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:45.442837 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.443754 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:45.444662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.445255 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:45.445809 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:45.446641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.447665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:45.447665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:45.447665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.447665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:45.447665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.447665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:45.449164 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:45.449524 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:45.449636 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:45.449751 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:45.450083 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:45.450233 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.450719 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:45.451448 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:45.452374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.453073 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.453721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:45.454298 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.455039 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:45.455943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.455943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:45.455943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:45.455943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.455943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:45.455943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.455943 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:45.456302 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:45.456586 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:45.456673 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:45.456794 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:45.457071 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:45.457306 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.457745 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:45.458461 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.458856 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.460140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:45.460492 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:45.460843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:45.461254 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.461732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:45.462249 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:45.462482 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:45.462588 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:45.462731 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:45.462993 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:45.463202 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:45.463333 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.463805 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:45.464607 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.464957 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:45.465321 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.465321 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:45.465701 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:45.466144 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:45.466670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:45.466670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:45.466670 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:49.981128 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:49.982491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:49.982491 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:49.982688 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:50.010236 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:50.010706 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.011588 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:50.012358 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.012757 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:50.013473 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:50.013959 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.014522 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:50.014522 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:50.014522 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.014522 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:50.014522 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.014522 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:50.441350 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:50.441699 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:50.441898 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:50.443764 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:50.444355 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.445785 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:50.447324 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.448033 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:50.448728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.448728 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:50.450310 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:50.451090 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:50.452020 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.452560 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:50.453391 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:50.454825 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.455565 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.456229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:50.459524 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.459839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:50.460183 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:50.460614 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:50.460994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:50.461442 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.462186 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:50.465685 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.466237 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:50.467360 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.467990 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:50.468572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:50.469196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:50.469872 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:50.470694 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:50.471276 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:50.471276 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:50.471276 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:50.471276 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.478354 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:50.478904 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.479283 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:50.479614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.479614 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:50.485651 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:50.486402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:50.487231 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:50.487231 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:50.487231 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:54.982513 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:54.983263 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:54.983263 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:54.984476 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:55.013797 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:55.014068 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.014699 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:55.015629 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.016252 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:55.016890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:55.017666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.018585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:55.018585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:55.018585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.018585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:55.018585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.018585 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:55.441994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:55.442420 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:55.442653 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:55.443727 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:55.444047 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.444902 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:55.446433 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.447042 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:55.447793 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.447793 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:55.448194 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:55.448920 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.449760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:55.449760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.449760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:55.450351 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:55.450532 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:55.450624 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:55.450768 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:00:55.451230 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:55.451584 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.452228 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:55.452971 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.453625 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.454026 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:55.454553 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:55.455133 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.455772 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.456374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:55.459907 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:55.460145 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:55.460327 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:55.460497 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:55.460942 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:55.461336 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.462007 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:00:55.462742 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.463153 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.463717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:55.464041 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:55.464424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.464424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:55.464424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.464424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.465406 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:55.465868 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:55.466419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.466419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:55.466419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.466419 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:55.469101 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:55.469446 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:55.469714 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:55.470028 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:00:55.470498 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:55.470821 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:00:55.470993 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.471779 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:00:55.472698 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.473036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:00:55.473402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.473402 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:00:55.473693 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:00:55.474115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:00:55.474618 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:00:55.474618 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:00:55.474618 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:00:59.983297 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:00:59.983987 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:00:59.984198 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:00:59.985911 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:00.015541 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:00.015908 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.016632 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:00.017477 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.018028 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:00.018751 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:00.019482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.020172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:00.020172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:00.020172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.020172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:00.020172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.020172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:00.442616 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:00.442995 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:00.443173 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:00.443973 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:00.444309 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.445150 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:00.446587 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.447107 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:00.447721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.447721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:00.448224 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:00.448930 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.449753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:00.449753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.449753 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:00.450688 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:00.451006 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:00.451177 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:00.451308 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:01:00.451736 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:00.452031 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.452759 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:00.453474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.453896 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.454284 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:00.454624 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:00.455016 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.455016 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:00.455016 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.455016 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:00.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.455712 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:00.456171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.456171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:00.456171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:00.456171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.456171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:00.456171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.456171 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:00.460184 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:00.460375 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:00.460497 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:00.460618 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:00.461018 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:00.461424 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.462056 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:00.462761 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.463167 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.463611 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:00.463916 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:00.464321 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.464321 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:00.464321 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.464321 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.464665 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:00.465134 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:00.465658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.465658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:00.465658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.465658 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:00.469389 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:00.469575 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:00.469679 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:00.469816 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:00.470185 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:00.470584 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:00.470755 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.471415 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:00.472223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.472619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:00.472956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.472956 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:00.473145 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:00.473973 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:00.474476 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:00.474476 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:00.474476 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:04.984536 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:04.985246 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:04.985337 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:04.986597 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:05.019244 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:05.019669 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.020603 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:05.021196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.021815 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:05.022382 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:05.023036 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.023885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:05.023885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:05.023885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.023885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:05.023885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.023885 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:05.443237 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:05.443957 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:05.444162 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:05.444974 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:05.445160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.445758 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:05.447408 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.447970 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:05.448534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.448534 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:05.449151 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:05.449890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.451053 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:05.451053 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.451053 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:05.451261 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:05.451656 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:05.451843 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:05.452038 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:01:05.452760 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:05.453093 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.453890 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:05.454709 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.455188 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.455541 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:05.455921 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:05.456305 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.456305 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:05.456305 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.456305 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:05.456581 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.457024 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:05.457603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.457603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:05.457603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:05.457603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.457603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:05.457603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.457603 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:05.460601 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:05.460892 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:05.461003 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:05.461130 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:05.461566 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:05.461907 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.462619 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:05.463359 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.463777 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.464196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:05.464552 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:05.464915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.464915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:05.464915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.464915 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.465274 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:05.465734 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:05.466229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.466229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:05.466229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.466229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:05.469768 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:05.470041 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:05.470166 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:05.470342 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:05.470786 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:05.471121 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:05.471339 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.472156 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:05.473630 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.473975 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:05.474320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.474320 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:05.475331 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:05.475763 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:05.476280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:05.476280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:05.476280 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:09.985093 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:09.985774 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:09.986054 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:09.987340 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:10.016881 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:10.017393 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.018275 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:10.018978 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.019871 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:10.020480 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:10.021059 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.021679 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:10.021679 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:10.021679 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.021679 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:10.021679 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.021679 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:10.443886 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:10.444394 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:10.444526 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:10.445332 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:10.445597 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.446286 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:10.447075 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.448370 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:10.452427 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:10.453228 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:10.453645 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.453832 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:10.457735 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:10.458482 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.459099 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.459737 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:10.460270 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.460982 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:10.461760 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:10.462195 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:10.462195 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:10.462399 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.463061 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:10.463798 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.464466 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.464870 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:10.465364 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:10.465944 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:10.466572 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.467383 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:10.470174 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:10.470693 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:10.470997 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:10.471430 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:10.472021 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:10.472560 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:10.472843 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.473732 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:10.474707 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.475115 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:10.475465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.475465 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:10.475867 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:10.477048 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:10.478001 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:10.478001 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:10.478001 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:14.986074 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:14.987041 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:14.987331 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:14.989049 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:15.010442 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG dbcounter [-] [105475] Writing DB stats neutron:SELECT=232 {{(pid=105475) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} May 22 01:01:15.022298 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:15.022493 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.023172 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:15.024818 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.025412 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:15.025931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.025931 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:15.026537 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:15.027201 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.027980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:15.027980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.027980 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:15.444394 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:15.446628 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:15.446711 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:15.447196 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:15.447368 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.447880 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:15.448566 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.449119 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:15.449699 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:15.450374 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.451180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:15.451180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:15.451180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.451180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:15.451180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.451180 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:15.452246 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:15.452504 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:15.452615 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:15.452615 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:01:15.452875 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:15.453008 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.453389 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:15.453989 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:15.454641 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.455277 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.455839 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:15.456567 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.457013 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:15.457531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.457531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:15.457531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:15.457531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.457531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:15.457531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.457531 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:15.461389 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:15.461684 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:15.461854 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:15.462039 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:15.462516 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:15.462855 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.463588 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:15.464741 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.465152 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.466164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:15.466829 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:15.467474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.467474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:15.467474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.467474 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.467925 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:15.468717 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:15.469516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.469516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:15.469516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.469516 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:15.470570 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:15.470774 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:15.470880 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:15.471040 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:15.471346 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:15.471570 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:15.471687 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.472164 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:15.473297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.473723 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:15.474141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:15.474689 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:15.475196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:15.475196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:15.475196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:15.475196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:15.475196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:15.475196 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:19.986994 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:19.987899 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:19.987899 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:19.988960 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:20.022868 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:20.023297 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.024329 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:20.025369 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.025937 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:20.026589 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:20.027376 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.029032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:20.029032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:20.029032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.029032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:20.029032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.029032 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:20.446972 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:20.447429 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:20.447623 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:20.448412 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-3c6124f8-a1b8-45a5-aeb3-a7ce3b464973, routes=[], if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:20.448662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.449229 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:20.450802 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.451338 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:20.451842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.451842 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:20.452714 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:20.453401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.454140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:20.454140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.454140 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:20.454140 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:20.454140 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:20.454140 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:20.454140 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting fip-distributed flag in NB_Global to False {{(pid=105475) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} May 22 01:01:20.456437 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:20.456599 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.456997 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:20.457721 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.458141 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.458660 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:20.459536 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:20.460273 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^ May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.460674 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.461160 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:20.461905 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:20.462014 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_ha_failover {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:20.462115 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:20.462263 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:20.462509 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:20.462666 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.463083 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() May 22 01:01:20.463799 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.command May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:20.464869 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:20.465662 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:20.466056 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute May 22 01:01:20.466401 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.466880 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics May 22 01:01:20.470960 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=105475) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} May 22 01:01:20.471175 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} May 22 01:01:20.471315 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=105475) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} May 22 01:01:20.471428 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=105475) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} May 22 01:01:20.473292 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:20.473292 np0000176101 neutron-ovn-maintenance-worker[105475]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=105475) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} May 22 01:01:20.473292 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Traceback (most recent call last): May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: txn.results.put(txn.do_commit()) May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: ^^^^^^^^^^^^^^^ May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: raise RuntimeError(msg) May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.473494 np0000176101 neutron-ovn-maintenance-worker[105475]: May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: May 22 01:01:20.474223 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.475088 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics [None req-26480c8c-0c74-4a1b-8ea8-8d8b405f3c5c None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics Traceback (most recent call last): May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics work() May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(*args, **kwargs) May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics return f(self, *args, **kwargs) May 22 01:01:20.475601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics result = f(*args, **kwargs) May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics next(self.gen) May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics with self.create_transaction( May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics self.result = self.commit() May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^ May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise result.ex May 22 01:01:20.476054 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run May 22 01:01:20.476601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics txn.results.put(txn.do_commit()) May 22 01:01:20.476601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ May 22 01:01:20.476601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit May 22 01:01:20.476601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics raise RuntimeError(msg) May 22 01:01:20.476601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it May 22 01:01:20.476601 np0000176101 neutron-ovn-maintenance-worker[105475]: ERROR futurist.periodics